PANCast Episode 21: Cortex XDR Agent Logs and Operational Status Analysis

[ozheng]

Episode Transcript:

John:

Hello, and welcome back to PANCast.

In today's episode, we're going to be talking about log analysis for Cortex XDR. We have a special guest today Daryl Mae who will share more on this topic.

Before we get started, Daryl Mae, could you tell us more about yourself?

Daryl Mae:

Hello John and hello to all PANCast listeners. I am Daryl Mae and a Senior Technical

Support Engineer from Cortex XDR. I’veDaryl Mae is a Senior Technical Support Engineer for Cortex XDR and has a vast experience in the Support environment. She is passionate about sharing her knowledge and expertise with customers, especially on Cortex XDR agent statuses. been supporting Cortex XDR customers for almost 2 years now and I am elated to share what I think is the most interesting topic for Cortex XDR and that is about Cortex XDR Agent Operational Status.

John:

Thank you, Daryl Mae. Could you tell us what is the Cortex Agent Operational Status?

Daryl Mae:

Sure John, so the XDR agent reports the operational status as Protected, Partially Protected, and Unprotected.

And these operational statuses will indicate whether the agent is providing protection according to your predefined security policies and profiles. And it’ll also help you to identify when an agent encounter any technical issues or misconfiguration that interferes the agent’s protection capability. 

And with this, I will discuss more on the agent operational status, and then I will also discuss how to collect and analyze endpoint logs.

John:

Great, can you tell us more about the various reports, their status and how they can be used?

Daryl Mae:

Yeah, absolutely, so for protected status, this indicates that the XDR agent is running as configured; for partially protected it indicates that the XDR agent reported one or more exceptions; and lastly, unprotected status indicates that the XDR agent is not enforcing protection on the endpoint. 

Well, I would like to emphasize that I will be covering agents that are in partially protected and unprotected states. 

So, when you get this report in your XDR server console, you’ll get an idea that something is not right. Then you start to wonder what might have happened to these endpoints or what has caused them to be in that state all of a sudden. You can then start to think about what has changed in the environment and so on.

And the first important thing to do, is to collect the agent logs. 

Now, it is commonly known for the fact that logs are vital part of any troubleshooting that gives us a clue of what's going on and tells us things. Though, logs analysis is a tedious task, right? However, this is essential and useful to analysts and Support Engineers to find the root cause of the issue.

John:

Looks like the logs can provide insights into what is happening. So how do we go about collecting these Agent Endpoint Logs?

How to Collect Agent Endpoint Logs?

Daryl Mae:

Yes, you are absolutely right, now, there are three ways where you can retrieve endpoints agent logs. That is through Cortex XDR console, from the Agent console on the endpoint that you can find in the system tray for Windows, and lastly using the cytool command which you can run in the command prompt or live terminal.

So, once you have generated the logs, you are now ready to investigate! So, what’s next?  Well, you need to unzip the folder and it will be extracted into different kinds of file logs and groups of folders. You will be looking at the endpoint log named trapsd which is in text document type and can be found in the log folder as well. 

By the way, you have the option to use any free file search tool for logs analysis, or you can just simply use windows notepad or any tools that you are familiar with.

John:

OK, so now that we have the logs, how do we make sense of what we collected?

What About Log Analysis?

Daryl Mae:

Alright, so now going back to the trapsd log, once you open it, you can then see bits and pieces of information about the agent’s telemetry, heartbeat, communication, and so on. So for the operational Status issue, let's say Partially Protected or Unprotected, you will be looking at the Agent’s telemetry specifically on the AgentOperationalStatusReporterThread. You can search this in your notepad, If you are using it, then you can just simply use the, you know, the search and find features and it will show you the results.

John: 

This is good information. Can you tell us more about the Cortex XDR components?

What are the Cortex XDR Components?

Daryl Mae:

Absolutely, these are the components that anyone should know. So, you will be expecting to see the Cortex XDR agent’s components status in the logs and these are:

• AntiexploitStatus
• AntimalwareStatus
• EDRStatus
• Fileprevalence
• hostfirewall
• and lastly General status.

So, an important note to know is that when the Agent is Enabled and Protected, this signifies no issues, and the status code will all show a “Zero”.

John:

I noticed that there are various codes relating to Operational Status. Could you share more details on these codes?

What are the different Agent Operational Status Codes?

Daryl Mae:

Glad you asked, I wanted to share that we have a various status codes for each platforms and those codes have a corresponding meaning. In Windows platforms, it can tell you if the agent is having a general failure, if the agent is not running or if the disk quota is exceeded.

And as for Linux, you can see things like unsupported kernel versions and if the agent is running asynchronously, and so on. 

Last but not the least, you can also identify if the Cortex XDR system extension requires Full Disk Access in a MacOS machines. 

So, these are just some information corresponding to the operational status codes and the lists of these codes can be found in our Knowledge Base article and you can absolutely refer to the transcript for more information.

John:

This provides a good overview of what the codes can tell us about their operational state.

Daryl Mae, what happens if the agent is in a disconnected status? What should we do?

What If the Agent is in Disconnected Status?

Daryl Mae:

We get this question a lot,  and I would like to share that if you seem to notice that a certain endpoint is disconnected or wondering if they have encountered connection issues, you can still do the steps I have mentioned before, and you can focus on the trapsd log and look for the communication log this time. You can simply search for the word “Error” and it will show various error messages such as - exception error, SSL error as well as  DNS errors and timeouts, too. 

So, with this initial step of the investigation, you can acquire quick information on your endpoint’s protection status and you can perform your initial troubleshooting. 

I think It’s not that difficult at all!

John:

It certainly helps that you have organized and shared a structure that we can work with. What will be the key takeaways from this episode?

Key Takeaways

Daryl Mae:

Alright, so we have discussed:

• The different kinds of Agent Operational Statuses
• How to collect endpoint logs and how to analyze them
• The lists of Cortex XDR Components
• And how to analyze Disconnected agent

John:

Thanks, Daryl Mae, for sharing these insights on collecting, analyzing and investigating endpoints logs. You can find the episode’s transcript and valuable links at live.paloaltonetworks.com under PANCast.

Daryl Mae:

Thank you for having me, John. I am honored to be able to share and discuss some tips on this topic. And I hope I can join you again in the next PANCast episode.

John:

PANCasters, if you have topics you need us to cover, please send in your feedback through the Ideas Submission page on LIVEcommunity and we’ll be happy to review them.

Until next time. Bye!

Related Content:

• Cortex XDR Supported Kernel Module Versions by Distribution
• Agent Operational Status Codes on Cortex XDR

[jnathan]

Thank you for sharing details on the log codes. It helps to deep dive and take action where needed. 

[wechoi]

Great information! Agent logs is indeed essential for troubleshooting agent health and would contain clues on issues that may be easily resolve by one and without the need of support. Thanks Daryl !