Cortex XDR Linux Agent: IPC error 107 caused by SELinux denying execution of pmd

[Robert.Michalek]

Cortex XDR Linux Agent – IPC error 107 / traps_pmd.service failed to start

Environment

• Palo Alto Networks Cortex XDR Agent for Linux

• Oracle Linux

• SELinux: Enforcing

• /opt: XFS

• /opt was mounted without noexec

Symptoms

After installing the Cortex XDR Agent, cytool status failed with:

Failed to get status summary data from the agent,
error: Ipc send message failed with error: 107:System

The Cortex XDR agent log showed:

Ipc client connected to channel:
/opt/traps/ipc/support.D277B246304A403F8641295EC28FBB2A

Executing method 'GetStatusSummary'

IpcClient: Failed to write header size. Error: system:107

Ipc send message failed with error: 107:System

****Cytool command [/opt/traps/bin/cytool status ] completed with error. Error code = 69****

At the same time, traps_pmd.service was failing:

traps_pmd.service: Failed at step EXEC spawning /opt/traps/bin/pmd: Permission denied

traps_pmd.service: Main process exited, code=exited, status=203/EXEC

traps_pmd.service: Failed at step EXEC spawning
/opt/traps/km_utils/km_manage: Permission denied

Initially, the IPC error 107 appeared to indicate a problem with communication between the Cortex XDR components.

Troubleshooting

The binaries had execute permissions:

ls -l /opt/traps/bin/pmd /opt/traps/km_utils/km_manage

Result:

-rwx------. 1 root root ... /opt/traps/bin/pmd
-rwx------. 1 root root ... /opt/traps/km_utils/km_manage

The /opt filesystem was checked to rule out noexec:

findmnt -T /opt/traps/bin/pmd -o TARGET,FSTYPE,OPTIONS

Result:

/opt xfs rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota

There was no noexec option.

SELinux was then checked:

getenforce

Result:

Enforcing

The decisive evidence came from the SELinux audit log:

type=AVC ... avc: denied { execute }
name="pmd"
scontext=system_u:system_r:init_t:s0
tcontext=unconfined_u:object_r:unlabeled_t:s0
tclass=file
permissive=0

The same occurred for km_manage:

type=AVC ... avc: denied { execute }
name="km_manage"
scontext=system_u:system_r:init_t:s0
tcontext=unconfined_u:object_r:unlabeled_t:s0
tclass=file
permissive=0

Checking the file contexts confirmed that the Cortex XDR installation under /opt/traps had been labelled unlabeled_t:

ls -lZ /opt/traps/bin/pmd

Initially:

-rwx------. root root unconfined_u:object_r:unlabeled_t:s0 ... /opt/traps/bin/pmd

The /opt/traps, /opt/traps/bin and /opt/traps/km_utils directories were also labelled unlabeled_t.

The expected SELinux context was checked with:

matchpathcon /opt/traps

which returned:

/opt/traps system_u:object_r:usr_t:s0

Root cause

The root cause was an incorrect SELinux file context on the Cortex XDR installation.

The Cortex XDR binaries under /opt/traps were labelled:

unlabeled_t

SELinux was running in Enforcing mode and therefore denied systemd permission to execute pmd and km_manage.

As a result:

1. traps_pmd.service could not execute pmd.

2. The Cortex XDR agent daemon did not start correctly.

3. cytool could connect to the IPC channel, but the backend could not properly process the request.

4. cytool status returned IPC error 107:

   Ipc send message failed with error: 107:System
   

5. cytool consequently returned error code 69.

Therefore, IPC error 107 was a secondary symptom of the Cortex XDR agent daemon not being able to start, rather than the primary cause.

Resolution

The SELinux contexts were restored according to the local SELinux policy:

restorecon -Rv /opt/traps

Afterward, the context of pmd changed from:

unconfined_u:object_r:unlabeled_t:s0

to:

unconfined_u:object_r:bin_t:s0

and km_manage changed to:

unconfined_u:object_r:usr_t:s0

The Cortex XDR service was then restarted:

systemctl restart traps_pmd.service

and verified with:

systemctl status traps_pmd.service

Finally, the agent status was checked again:

/opt/traps/bin/cytool status

Conclusion

The initial error reported by the Cortex XDR tooling was:

Ipc send message failed with error: 107:System

However, investigation showed that the underlying problem was an SELinux execution denial.

The key diagnostic evidence was:

avc: denied { execute }
tcontext=unconfined_u:object_r:unlabeled_t:s0

The problem was resolved by restoring the SELinux contexts for /opt/traps:

restorecon -Rv /opt/traps

This restored the expected SELinux labelling and allowed the Cortex XDR agent components to execute normally.