EPM Agent with excessive network consumption

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements

Welcome Idira to LIVEcommunity!

Our community just got stronger! Explore new identity security resources, connect with experts, and discover more ways to learn and collaborate.

Meet Idira

EPM Agent with excessive network consumption

L3 Networker

One of my users has seen that their process com.cyberark.CyberArkEPMNetworkSession sends a large amount of data (about 700 GB in 2 weeks and 15 GB on 2 hours). Which requires the EPM agent to be reinstalled.

 

Has anyone seen something similar to this?Field_Content_versions_oct20__068Vy000012I4DCIA0.png 

 

1 accepted solution

Accepted Solutions

Hello we have a full on Dev environment and our some of our Devs have noticed and some on demand that cyebrark epm or generally speaking there is a massive network bandwidth usage issue with the default epm package, 2 things seems to resolve the issue, either moving a user to a fresh set with no epm policies , this could be due to too many audits causing the high bandwidth, or disabling the network content filter, which we did for 5 devs and they have since not reported any issues with high bandwidth usage by their ISP or when at work and we're talking some users uploading 100's of GB a day or even 1 TB over 24 hours, case number Case Number

04114692

I know of 1 other user on the cyberark discord who reported the issue too but he's a MSP provider, we're going as far as removing the content filter for all our devs this weekend, however we do have a full cyber security stack. Theory was other tools perform the same actionand loop over each other but the simple fact is this we're going ahead and repackaging the epm macos agent without content filter, I do personally think we have way too many allow policies that are audited possibly but we're not seeing the same problem on our windows machines and it could still be a conflict between the content filter and another cyber product - https://community.cyberark.com/s/article/EPM-macOS-Purpose-of-CyberArk-content-filtering - I must admit we do also see very high disc usage but that's even without epm installed and mutual exclusions nd log files were checked from all the other cyber tools but i don't personally think they really have them setup properly and I dare say the SSD are being thrashed daily, anyway you can see below once the content filter was disabled for the user his ISP bandwidth usage dropped in the yellow bar back to norma/. https://community.cyberark.com/s/article/EPM-macOS-Purpose-of-CyberArk-content-filtering The only really frustrating part is these 5 devs are still in the normal MACOS set with the full auditing is still occurring so how can it be an auditing issue, unless its an policy auditing issue with the content filter itselfField_Content_versions_oct20__068Vy000012Mz9kIAC.pngField_Content_versions_oct20__068Vy000012MyqRIAS.pngField_Content_versions_oct20__068Vy000012NSQ9IAO.pnghttps://community.cyberark.com/s/article/EPM-macOS-Purpose-of-CyberArk-content-filtering - we also had a report of 1 user stating that his RAM usage was about 10GB for one of the cyberark processes, I will double check if removing the content filter resoled that issue, not heard back from him so that must have fixed it.

 

we did run fs_usage, systemextensionsctl list, fs_monitor tool and i would like to use https://github.com/Brandon7CC/mac-monitor to try and figure out the issue if you could possibly link my case to yours that would be nice and we did double check Mutual exclusions and uninstalled the other cyber tools to rule them out , issue is also existing on 25.11 with the default package. weirdly Cyberark did come back and say the "Please note that our network extension only monitors traffic; it does not take any action unless a policy explicitly depends on network connectivity (for example, a conditional policy requiring a connection to a specific resource).

 

The network session handles tasks such as:

- Registering with the server

- Updating policies

- Maintaining connections to our servers

 

When the network extension is enabled, it simply monitors activity. You will see traffic passing through the filter and then out."

 

And I heavily troubleshooted with my limited rights its not like I have access to the other cyber tools and need them to check the portals for any issues . for sure the average was 20GB to 40Gb a day but some users on demand could cause the issue such as accessing github. one user uploaded 3TB to his ISP over a 3 day weekend without even using the macos he simply left it switched on.

View solution in original post

7 REPLIES 7

L7 Applicator

Hi @Ryan Kuczer​ 

 

Are you using the latest EPM agent for MacOS? If not please use the latest version.

 

Please refer to these documents / KBs.

 

https://docs.cyberark.com/epm/latest/en/content/intro/macos-bp-solutions.htm

 

https://community.cyberark.com/s/article/EPM-User-is-experiencing-slowness-while-the-EPM-Agent-is-installed-on-the-endpoint

 

https://community.cyberark.com/s/article/EPM-Performance-problems-when-Anti-Tampering-Protection-conflicts-with-another-security-product

 

 

Hope it helps!

Hello we have a full on Dev environment and our some of our Devs have noticed and some on demand that cyebrark epm or generally speaking there is a massive network bandwidth usage issue with the default epm package, 2 things seems to resolve the issue, either moving a user to a fresh set with no epm policies , this could be due to too many audits causing the high bandwidth, or disabling the network content filter, which we did for 5 devs and they have since not reported any issues with high bandwidth usage by their ISP or when at work and we're talking some users uploading 100's of GB a day or even 1 TB over 24 hours, case number Case Number

04114692

I know of 1 other user on the cyberark discord who reported the issue too but he's a MSP provider, we're going as far as removing the content filter for all our devs this weekend, however we do have a full cyber security stack. Theory was other tools perform the same actionand loop over each other but the simple fact is this we're going ahead and repackaging the epm macos agent without content filter, I do personally think we have way too many allow policies that are audited possibly but we're not seeing the same problem on our windows machines and it could still be a conflict between the content filter and another cyber product - https://community.cyberark.com/s/article/EPM-macOS-Purpose-of-CyberArk-content-filtering - I must admit we do also see very high disc usage but that's even without epm installed and mutual exclusions nd log files were checked from all the other cyber tools but i don't personally think they really have them setup properly and I dare say the SSD are being thrashed daily, anyway you can see below once the content filter was disabled for the user his ISP bandwidth usage dropped in the yellow bar back to norma/. https://community.cyberark.com/s/article/EPM-macOS-Purpose-of-CyberArk-content-filtering The only really frustrating part is these 5 devs are still in the normal MACOS set with the full auditing is still occurring so how can it be an auditing issue, unless its an policy auditing issue with the content filter itselfField_Content_versions_oct20__068Vy000012Mz9kIAC.pngField_Content_versions_oct20__068Vy000012MyqRIAS.pngField_Content_versions_oct20__068Vy000012NSQ9IAO.pnghttps://community.cyberark.com/s/article/EPM-macOS-Purpose-of-CyberArk-content-filtering - we also had a report of 1 user stating that his RAM usage was about 10GB for one of the cyberark processes, I will double check if removing the content filter resoled that issue, not heard back from him so that must have fixed it.

 

we did run fs_usage, systemextensionsctl list, fs_monitor tool and i would like to use https://github.com/Brandon7CC/mac-monitor to try and figure out the issue if you could possibly link my case to yours that would be nice and we did double check Mutual exclusions and uninstalled the other cyber tools to rule them out , issue is also existing on 25.11 with the default package. weirdly Cyberark did come back and say the "Please note that our network extension only monitors traffic; it does not take any action unless a policy explicitly depends on network connectivity (for example, a conditional policy requiring a connection to a specific resource).

 

The network session handles tasks such as:

- Registering with the server

- Updating policies

- Maintaining connections to our servers

 

When the network extension is enabled, it simply monitors activity. You will see traffic passing through the filter and then out."

 

And I heavily troubleshooted with my limited rights its not like I have access to the other cyber tools and need them to check the portals for any issues . for sure the average was 20GB to 40Gb a day but some users on demand could cause the issue such as accessing github. one user uploaded 3TB to his ISP over a 3 day weekend without even using the macos he simply left it switched on.

Hello we have a full on Dev environment and our some of our Devs have noticed and some on demand that cyebrark epm or generally speaking there is a massive network bandwidth usage issue with the default epm package, 2 things seems to resolve the issue, either moving a user to a fresh set with no epm policies , this could be due to too many audits causing the high bandwidth, or disabling the network content filter, which we did for 5 devs and they have since not reported any issues with high bandwidth usage by their ISP or when at work and we're talking some users uploading 100's of GB a day or even 1 TB over 24 hours, case number Case Number

04114692

I know of 1 other user on the cyberark discord who reported the issue too but he's a MSP provider, we're going as far as removing the content filter for all our devs this weekend, however we do have a full cyber security stack. Theory was other tools perform the same actionand loop over each other but the simple fact is this we're going ahead and repackaging the epm macos agent without content filter, I do personally think we have way too many allow policies that are audited possibly but we're not seeing the same problem on our windows machines and it could still be a conflict between the content filter and another cyber product - https://community.cyberark.com/s/article/EPM-macOS-Purpose-of-CyberArk-content-filtering - I must admit we do also see very high disc usage but that's even without epm installed and mutual exclusions nd log files were checked from all the other cyber tools but i don't personally think they really have them setup properly and I dare say the SSD are being thrashed daily, anyway you can see below once the content filter was disabled for the user his ISP bandwidth usage dropped in the yellow bar back to norma/. https://community.cyberark.com/s/article/EPM-macOS-Purpose-of-CyberArk-content-filtering The only really frustrating part is these 5 devs are still in the normal MACOS set with the full auditing is still occurring so how can it be an auditing issue, unless its an policy auditing issue with the content filter itselfField_Content_versions_oct20__068Vy000012Mz9kIAC.pngField_Content_versions_oct20__068Vy000012MyqRIAS.pngField_Content_versions_oct20__068Vy000012NSQ9IAO.pnghttps://community.cyberark.com/s/article/EPM-macOS-Purpose-of-CyberArk-content-filtering - we also had a report of 1 user stating that his RAM usage was about 10GB for one of the cyberark processes, I will double check if removing the content filter resoled that issue, not heard back from him so that must have fixed it.

 

we did run fs_usage, systemextensionsctl list, fs_monitor tool and i would like to use https://github.com/Brandon7CC/mac-monitor to try and figure out the issue if you could possibly link my case to yours that would be nice and we did double check Mutual exclusions and uninstalled the other cyber tools to rule them out , issue is also existing on 25.11 with the default package. weirdly Cyberark did come back and say the "Please note that our network extension only monitors traffic; it does not take any action unless a policy explicitly depends on network connectivity (for example, a conditional policy requiring a connection to a specific resource).

 

The network session handles tasks such as:

- Registering with the server

- Updating policies

- Maintaining connections to our servers

 

When the network extension is enabled, it simply monitors activity. You will see traffic passing through the filter and then out."

 

And I heavily troubleshooted with my limited rights its not like I have access to the other cyber tools and need them to check the portals for any issues . for sure the average was 20GB to 40Gb a day but some users on demand could cause the issue such as accessing github. one user uploaded 3TB to his ISP over a 3 day weekend without even using the macos he simply left it switched on.

and reinstalling the agent with the same settings you're eventually see the issue come back and it's worth asking some other devs to type uptime in terminal/iterm and take a screen shot of their network bandwidth usage tab in activity manager, i am sure you have the same problem as me and maybe look at your proxy tools/cyber tools logs and events and see if they're conflicting with each other ?My colleague's had this theory too "“The issue is reproducible, binary, and fully correlated with the content-filtering extension. This is not about endpoint session management. This is a deterministic increase of several orders of magnitude that disappears when the extension is removed. This qualifies as a defect in the NEFilterDataProvider / NEFilterControlProvider implementation or packet handoff loop, not a configuration issue.”

 Also add to the ticket the output from one of the affected hosts for: "systemextensionsctl list" command

 it sould be something like this:

  All NEFilter providers (Microsoft Defender, Zscaler, CyberArk) participate in the same pipeline, and any re-injection or mis-handling from one provider will create a loop across the chain. What we see is consistent with an NEFilter classification loop, at leas from my point of view. - it's a copout for my cyber to remove this function personally but it does resolve the issue in regards to content filter, but I'd also keep an eye on your IOPS disc usage , I know cyberark made a nugget on tracking this on youtube.

 

 

Given the reproducibility, and the binary behaviour with/without the CyberArk extension, this requires escalation to R&D for investigation of the EPM Network Extension (likely DataProvider/ControlProvider logic). This is not a configuration issue.

 "

and reinstalling the agent with the same settings you're eventually see the issue come back and it's worth asking some other devs to type uptime in terminal/iterm and take a screen shot of their network bandwidth usage tab in activity manager, i am sure you have the same problem as me and maybe look at your proxy tools/cyber tools logs and events and see if they're conflicting with each other ?My colleague's had this theory too "“The issue is reproducible, binary, and fully correlated with the content-filtering extension. This is not about endpoint session management. This is a deterministic increase of several orders of magnitude that disappears when the extension is removed. This qualifies as a defect in the NEFilterDataProvider / NEFilterControlProvider implementation or packet handoff loop, not a configuration issue.”

 Also add to the ticket the output from one of the affected hosts for: "systemextensionsctl list" command

 it sould be something like this:

  All NEFilter providers (Microsoft Defender, Zscaler, CyberArk) participate in the same pipeline, and any re-injection or mis-handling from one provider will create a loop across the chain. What we see is consistent with an NEFilter classification loop, at leas from my point of view. - it's a copout for my cyber to remove this function personally but it does resolve the issue in regards to content filter, but I'd also keep an eye on your IOPS disc usage , I know cyberark made a nugget on tracking this on youtube.

 

 

Given the reproducibility, and the binary behaviour with/without the CyberArk extension, this requires escalation to R&D for investigation of the EPM Network Extension (likely DataProvider/ControlProvider logic). This is not a configuration issue.

 "

L3 Networker

@Malhar Vora​ - thank you for those links, have read them thoroughly.

@Touseef-ur rehman​ - Thank you for the in-depth response I appreciate it a lot. This was exactly what I was looking for, so far I have reinstalled the agent and the dev users are not experiencing issues at the moment. Will follow this if the issue does occur again, but I am guessing based on what you said it will happen

  • 1 accepted solution
  • 129 Views
  • 7 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!