The Kubernetes auditing system tracks the activities of users, administrators, and other components impacting the cluster. Once you configure the Prisma Cloud CWP Kubernetes auditing feature, Prisma Cloud can ingest, analyze, and alert on security-relevant events. You can either write custom rules or use pre-written rules from Prisma Cloud Labs to evaluate the incoming audit stream and detect suspicious activities.
This article outlines troubleshooting steps to follow if audit logs are not visible in the console after configuring Kubernetes auditing for your Elastic Kubernetes Service (EKS).
First, confirm that the Kubernetes auditing configuration is successful. Navigate to the console and go to Defend > Access > Kubernetes to verify if an audit has been created.
Figure 1: Kubernetes-Audits_PaloAltoNetworks
If no audit is visible, wait 10-15 minutes. Prisma Cloud retrieves audits from AWS CloudWatch, polling it every 10-15 minutes for new data. If you still don’t see any events, follow the troubleshooting steps below:
Ensure that EKS is set up to export logs to AWS CloudWatch. Follow the instructions in the AWS documentation and make sure the EKS Control Plane Logging for "Audit" logs is enabled for your EKS cluster.
Verify that the AWS IAM Role or User configured for EKS Auditing has the necessary read-only permissions for CloudWatch. The role should have the following permissions:
logs:GetQueryResults
logs:StartQuery
logs:DescribeLogGroups
Refer to the AWS managed policy documentation for details on the required permissions.
If you’ve confirmed that CloudWatch permissions are correctly configured and still do not see audits in Prisma Cloud, review the CloudTrail logs. Check if events related to StartQuery, GetQueryResults, and DescribeLogGroups are logged from the user configured in Prisma Cloud.
For guidance, refer to the AWS CloudTrail documentation.
Figure 2: CloudTrails-Events_PaloAltoNetworks
If issues persist, review the console debug logs in Prisma Cloud. Search for the keyword “audit” and examine related errors. For more information, refer to the Prisma Cloud debug logs guide.
Figure 3: Console-Debuglogs_PaloAltoNetworks
Ensure that all onboarded clusters in the Prisma Cloud Kubernetes Auditing configuration page are active and properly integrated with Prisma Cloud. Inactive or incorrectly integrated clusters can prevent Prisma Cloud from polling audit logs effectively on the newly on-boarded clusters.
Figure 4: KubernetesAuditing-Configuration_PaloAltoNetworks
If, after following these troubleshooting steps, the issue remains unresolved, consult AWS and Prisma Cloud Support for further assistance:
AWS Support: Reach out for help with EKS and CloudWatch integration problems.
Prisma Cloud Support: Contact for assistance with issues related to Prisma Cloud’s integration with EKS.
This article outlines troubleshooting steps to follow if audit logs are not visible in the console after configuring Kubernetes auditing for your Elastic Kubernetes Service (EKS). By following these troubleshooting steps, you should see Prisma Cloud ingesting and alerting on security-relevant events for EKS.
[1] Prisma Kubernetes Auditing
[3] CloudWatch Logs Read-Only Policy
[4] Cloud Trail Logs
Kiran Kaukuntla is a senior customer success engineer specializing in Prisma Cloud, AWS, Azure, GCP, containers and Kubernetes. He uses collaborative approaches to break down complex problems into solutions for global enterprise customers and leverage their multi industry knowledge to inspire success.
