Cortex XDR Linux Agent: IPC error 107 caused by SELinux denying execution of pmd

[Robert.Michalek]

Cortex XDR Linux Agent – IPC error 107 / traps_pmd.service failed to start

Environment

• Palo Alto Networks Cortex XDR Agent for Linux

• Oracle Linux

• SELinux: Enforcing

• /opt: XFS

• /opt was mounted without noexec

Symptoms

After installing the Cortex XDR Agent, cytool status failed with:

Failed to get status summary data from the agent,
error: Ipc send message failed with error: 107:System

The Cortex XDR agent log showed:

Ipc client connected to channel:
/opt/traps/ipc/support.D277B246304A403F8641295EC28FBB2A

Executing method 'GetStatusSummary'

IpcClient: Failed to write header size. Error: system:107

Ipc send message failed with error: 107:System

****Cytool command [/opt/traps/bin/cytool status ] completed with error. Error code = 69****

At the same time, traps_pmd.service was failing:

traps_pmd.service: Failed at step EXEC spawning /opt/traps/bin/pmd: Permission denied

traps_pmd.service: Main process exited, code=exited, status=203/EXEC

traps_pmd.service: Failed at step EXEC spawning
/opt/traps/km_utils/km_manage: Permission denied

Initially, the IPC error 107 appeared to indicate a problem with communication between the Cortex XDR components.

Troubleshooting

The binaries had execute permissions:

ls -l /opt/traps/bin/pmd /opt/traps/km_utils/km_manage

Result:

-rwx------. 1 root root ... /opt/traps/bin/pmd
-rwx------. 1 root root ... /opt/traps/km_utils/km_manage

The /opt filesystem was checked to rule out noexec:

findmnt -T /opt/traps/bin/pmd -o TARGET,FSTYPE,OPTIONS

Result:

/opt xfs rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota

There was no noexec option.

SELinux was then checked:

getenforce

Result:

Enforcing

The decisive evidence came from the SELinux audit log:

type=AVC ... avc: denied { execute }
name="pmd"
scontext=system_u:system_r:init_t:s0
tcontext=unconfined_u:object_r:unlabeled_t:s0
tclass=file
permissive=0

The same occurred for km_manage:

type=AVC ... avc: denied { execute }
name="km_manage"
scontext=system_u:system_r:init_t:s0
tcontext=unconfined_u:object_r:unlabeled_t:s0
tclass=file
permissive=0

Checking the file contexts confirmed that the Cortex XDR installation under /opt/traps had been labelled unlabeled_t:

ls -lZ /opt/traps/bin/pmd

Initially:

-rwx------. root root unconfined_u:object_r:unlabeled_t:s0 ... /opt/traps/bin/pmd

The /opt/traps, /opt/traps/bin and /opt/traps/km_utils directories were also labelled unlabeled_t.

The expected SELinux context was checked with:

matchpathcon /opt/traps

which returned:

/opt/traps system_u:object_r:usr_t:s0

Root cause

The root cause was an incorrect SELinux file context on the Cortex XDR installation.

The Cortex XDR binaries under /opt/traps were labelled:

unlabeled_t

SELinux was running in Enforcing mode and therefore denied systemd permission to execute pmd and km_manage.

As a result:

1. traps_pmd.service could not execute pmd.

2. The Cortex XDR agent daemon did not start correctly.

3. cytool could connect to the IPC channel, but the backend could not properly process the request.

4. cytool status returned IPC error 107:

   Ipc send message failed with error: 107:System
   

5. cytool consequently returned error code 69.

Therefore, IPC error 107 was a secondary symptom of the Cortex XDR agent daemon not being able to start, rather than the primary cause.

Resolution

The SELinux contexts were restored according to the local SELinux policy:

restorecon -Rv /opt/traps

Afterward, the context of pmd changed from:

unconfined_u:object_r:unlabeled_t:s0

to:

unconfined_u:object_r:bin_t:s0

and km_manage changed to:

unconfined_u:object_r:usr_t:s0

The Cortex XDR service was then restarted:

systemctl restart traps_pmd.service

and verified with:

systemctl status traps_pmd.service

Finally, the agent status was checked again:

/opt/traps/bin/cytool status

Conclusion

The initial error reported by the Cortex XDR tooling was:

Ipc send message failed with error: 107:System

However, investigation showed that the underlying problem was an SELinux execution denial.

The key diagnostic evidence was:

avc: denied { execute }
tcontext=unconfined_u:object_r:unlabeled_t:s0

The problem was resolved by restoring the SELinux contexts for /opt/traps:

restorecon -Rv /opt/traps

This restored the expected SELinux labelling and allowed the Cortex XDR agent components to execute normally.

[susekar]

Hello @Robert.Michalek ,

Greetings for the day.

The diagnostic findings and root cause analysis in your report are confirmed and align directly with documented Cortex XDR behavior on Linux systems running SELinux in Enforcing mode.

-To remediate this issue and verify it, restore the SELinux security contexts across the Cortex XDR directory tree and restart the service.

Restore SELinux File Contexts:

Apply the system's default SELinux labeling policy to /opt/traps: restorecon -Rv /opt/traps

Note: If the agent was installed in a custom target path outside /opt/traps, run restorecon directly against that target directory, as restorecon does not traverse symbolic links.

The agent was installed in a non-default location (/cortex/install/traps), while /opt/traps was a symbolic link, confirmed by running ls -l /opt/traps. The initial restorecon -Rv /opt/traps attempt failed because restorecon does not follow symbolic links.

Restart the Agent Service:

Restart the systemd service managing the agent daemon: systemctl restart traps_pmd.service

------------------------------

Verify Service and Context Status:

-Check that the systemd service is active and running: systemctl status traps_pmd.service

-Verify that the agent binaries now display trusted contexts, such as bin_t or usr_t, rather than unlabeled_t: ls -Z /opt/traps/bin/pmd

-The observed SELinux context was: system_u:object_r:unlabeled_t:s0

-The expected context should be a trusted type, such as: unconfined_u:object_r:usr_t:s0or  system_u:object_r:default_t:s0

-Query the operational status of the agent components: /opt/traps/bin/cytool runtime query

-When correctly labeled under a targeted policy, the pmd process executes properly within the unrestricted service domain, clearing the IPC failure.

If you feel this has answered your query, please let us know by clicking like and on "mark this as a Solution".

Thanks & Regards,
S. Subashkar Sekar