- Access exclusive content
- Connect with peers
- Share your expertise
- Find support resources
Enhanced Security Measures in Place: To ensure a safer experience, we’ve implemented additional, temporary security measures for all users.
11-02-2022 08:51 AM - edited 11-02-2022 08:53 AM
So my co-worker has implemented some containerized solution and deployed it to EKS.
I used my access key to create a defender.yaml for him.
twistcli defender export kubernetes \
--address ${PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR} \
--user ${PRISMA_ADMIN_USER} \
--password ${PRISMA_SECRET} \
--cluster-address ${PRISMA_CLOUD_COMPUTE_SVC_ADDR}
And she deployed it.
kubectl create -f defender.yaml
When I visit her AWS account EKS Console, I can see the cluster and the daemon but is it healthy?
Why doesn't that daemon set appear in Manage/Defenders/Daemon Sets?
How can I determine the health of the daemon set?
Or, find the root-cause of it not reporting status?
11-16-2022 07:43 AM
So, here is how I solved it...
first, let's talk concepts...
EKS’ Clusters may be configured with Fargate Profile(s) and/or EC2 Managed Node Groups. These two kinds of EKS compute are separated and isolated. EC2 Nodes managed by customer; Fargate nodes managed by AWS. Because of this separation, a cluster daemonset cannot not protect Fargate tasks.
IMO, those concepts are poorly communicated here... https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/defender_ty...
Prisma Defenders may be deployed to either or both types of EKS compute in the same cluster.
EKS on Fargate – a defender for this type of compute are embedded into the image at “build” time. Each image has its own defender. To configure, see https://wiki.comp.pge.com/display/CCE/Embed+Defender+into+a+Container+for+EKS+on+Fargate. You can confirm the defenders are protecting the task by navigating to either PCC/Manage/Defenders/Manage/Defenders and/or PCC/Monitor/Runtime/App-Embedded observations.
EKS on EC2 – defenders are deployed as Kubernetes daemon sets. Given the daemonset AND a managed node group of EC2s are deployed then you can confirm one twistlock/defender pod per node. To confirm that the EKS Cluster is protected, navigate to PCC/Manage/Cloud Accounts; filter on your account number then choose the Action, Account Details, to see the Cloud Discovery results rendering which clusters are Defended or not. To configure, improve this document here https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_def...
also this document too needs to be corrected... https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_kub...
Bottom-line: The resolution to my original issue was that the daemonset.yaml produced by the command that I gave was missing two options, CRI and privileged. The daemon must use Container Runtime Interface and it MUST run privileged for it to protect the other pods' containers.
The command that produces a defect-free daemonset, defender.yaml is...
export PRISMA_KEY_ID=SECRET
export PRISMA_SECRET=SECRET
export PRISMA_CONSOLE=https://us-east1.cloud.twistlock.com/SECRET
export PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR=https://us-east1.cloud.twistlock.com/SECRET
export PRISMA_CLOUD_COMPUTE_SVC_ADDR=us-east1.cloud.twistlock.com
twistcli defender export kubernetes \
--user ${PRISMA_KEY_ID} \
--password ${PRISMA_SECRET} \
--address ${PRISMA_CLOUD_COMPUTE_CONSOLE_API_ADDR} \
--cluster-address ${PRISMA_CLOUD_COMPUTE_SVC_ADDR} \
--namespace twistlock \
--privileged \
--cri
11-02-2022 09:43 AM
.Tommy,
From the portal you should be able to view the defenders under Manage - Manage Defenders. Please see the screen shot below. I also noticed that you have two other DaemonSets configured but they do not have any pods running. Please check the health of the cluster, as well as run a kubectl get daemonsets --all-namespaces. You should have pods running under desired, current and ready. If you need more info on K8s daemonsets please review the links below, but it looks like the pods are not running at all. If you need help with a sample daemonset I can send you a yaml example to test.
https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
11-02-2022 11:48 AM
Thanks @MDavis29 I will read the docs to learn.
So I shared how I created my defender.yaml in the original post.
I understood that defender.yaml created with twistcli is good to deploy without change.
why do you offer to help with a sample daemonset yaml file?
am i missing something? just trying to learn.
11-02-2022 01:50 PM
Tommy,
My apologies if I confused you. The reason why I mentioned the sample daemonset was more for troubleshooting. With k8s is can be easier to remove some layers of complexity and use smaller deployments or DaemonSets. Looking at the screenshot you supplied of the workloads in the cluster, you can see that none of the pods for any of the DaemonSets are running. We need to identify why? this is most likely why info is not showing up in Prisma. Please run a kubectl describe on the pods in the DaemonSet and send the output, it should provide some clues as to what the problem is.
11-02-2022 02:13 PM
Tommy,
Checking the pods running in the DaemonSet is the first place to start. Then perform a describe on any pods that are not running.
11-02-2022 06:08 PM
Hello Tommy,
Can you please share yaml being used to deploy Prisma Defender?
Make sure to remove any personal identifiable information. The reason I am asking for yaml to look to see if all the relevant flags are turned on.
Regards,
11-03-2022 07:37 AM
Good morning @USheikh , attaching defender.yaml stripped of PEM data. The portal doesn't allow yaml so I zipped it up.
11-03-2022 11:39 AM
@MDavis29 Thanks for the advice. Ya know I can't surface ANY pods related to ANY of the daemon sets.
$ kubectl get daemonsets --all-namespaces
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system aws-node 0 0 0 0 0 <none> 42m
kube-system kube-proxy 0 0 0 0 0 <none> 42m
twistlock twistlock-defender-ds 0 0 0 0 0 <none> 18m
There are some pods running but I can't corrrelate them to the twistlock daemonset.
$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-load-balancer-controller-5fbf77b8ff-drnk5 1/1 Running 0 38m
kube-system aws-load-balancer-controller-5fbf77b8ff-lr4d6 1/1 Running 0 38m
kube-system cni-metrics-helper-dcccc58d4-465ln 1/1 Running 4 (3m1s ago) 25m
kube-system coredns-697b6f9fb8-2xnbb 1/1 Running 0 14m
kube-system coredns-697b6f9fb8-pw28w 1/1 Running 0 14m
I can't do a "kubectl describe pod" because there are no pods associated with the twistlock daemon set.
I am in the process of deploying this microservice, https://www.eksworkshop.com/beginner/050_deploy/, hopefully that gives the cluster some work. And then, I believe that the daemon sets will have pods, am i mistaken?
What do you suggest?
11-03-2022 02:06 PM - edited 11-04-2022 11:35 AM
Hello TommyHunt,
Looking at the yaml file you do not have "Nodes use Container Runtime Interface (CRI), not Docker" flag on.
Which version of EKS are you running? Is it running on daesmonset or CRI?
If it is CRI, then "Nodes use Container Runtime Interface (CRI), not Docker" flag needs to be turned on.
11-04-2022 08:06 AM - edited 11-08-2022 07:19 AM
Kubernetes version 1.23
I understand this cluster is intended to run Fargate Tasks on EKS so I believe that Daemon set is the correct defender type.
IF "it" means defender daemon set then yes there is a "defender" daemonset but we know this from an earlier message.
I am still learning; pronouns may create confusion rather than clarity.
Can you please use proper nouns instead of pronouns?
There exist pods.
For some reason, I understood that the cluster requires some workloads so that pods are created with the defender daemon.
So, I deployed the k8s-dashboard and an AWS Labs microservice, ecsdemo.
But notice that none of my workloads, aka deployments, have the "Ready" state, all 0s just like the daemon sets.
Due to cybersecurity policies, we don't configure clusters with public network interface, no public internet ingress.
Maybe that's why these workloads won't run.
Based on what you guys have shared.
I understand that since there is no deployment running then there is no pod running with a daemonset.
@USheikh I cannot correlate your reference to "Nodes use Container Runtime Interface (CRI), not Docker" to any element in the yaml file.
Can you provide yaml path of the element that you find unacceptable?
11-08-2022 11:27 AM
Hello TommyHunt!
Sorry I was OOO yesterday, please allow me time to review/test the scenario, and get back to you.
Regards,
11-08-2022 04:46 PM
Hello TommyHunt,
If I am not wrong, you are using yaml file to deploy a daemonset defender on EKS v1.23 that is running on Fargate, right?
Regards,
11-09-2022 06:44 AM
Thanks @MDavis29 .
so we added a managed node group, pods are now running.
I can see audit, garbagecollector and clusterip assignment CloudWatch log events...
But Prisma Cloud discovery reports the cluster as Defended=FALSE.
I can see that the k8s service, twistlock/defender, is allocated an IP, and some garbage collection but no trace of dameonset api calls or errors. Where do i go from here?
11-09-2022 08:04 AM
11-10-2022 02:47 PM
Hello TommyHunt,
Based on internal discussion, in order to install defender on fargate running eks v1.23, you would need to install app-embedded defender. Here is the documentation on app-embedded defender:
However, if you want to continue using the yaml method, please open a support case with us, so we can further investigate this and provide you with more details.
Regards,
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!