How do I enable PSMP debugging?
Steps
Note: Before restarting SSHD daemon, please make sure you have console access to PSMP server, in case the SSHD daemon cannot start due to configuration issues.
In the PVWA GUI:
1. Go to Administration → Options → Privileged Session Management → General Settings.
2. Under Server Settings set TraceLevels=1,2,3,4,5,6,7
3. Under Connection Client Settings set TraceLevels=1,2
Make sure to Apply or Save.
On the PSMP server:
1. Edit the /etc/ssh/sshd_config and add the following to the end of the file:
LogLevel DEBUG3
2. (v13.2 or earlier) If you installed PSMP with InstallCyberArkSSHD=yes, also add the following:
Note: If installed with "integrated" or "no", adding the following will cause the sshd service to fail to (re)start
PSMP_OpenSSHTraceLevels 1,2
PSMP_OpenSSHLogFolder /var/opt/CARKpsmp/logs/components
3. Restart the SSH Daemon to make these changes take effect:
Note: The method of restarting a service is OS version, and distribution dependent; Listed are a few examples:
/etc/init.d/sshd restart
service sshd restart
systemctl restart sshd
4. Run the following commands:
sshd -T >/etc/ssh/SshdRunningConfig
If OS is RHEL:
dnf list installed > installedpackages
If OS is Ubuntu:
apt list --installed > installedpackages
5. Reproduce the issue, then gather logs using one of the two following methods:
Automatically:
Utilizing the PSMPAssistant tool, follow step 3 in the following article: https://community.cyberark.com/s/article/PSMPAssistant
Manually:
The full contents of the following directories:
/var/opt/CARKpsmp/logs/
/var/opt/CARKpsmp/logs/components/
/var/opt/CARKpsmp/temp/
/var/opt/CARKpsmp/temp/old/
/etc/ssh/sshd.d/ (If RHEL 9.x)
The following files:
/etc/opt/CARKpsmp/basic_psmpserver.conf
/var/tmp/psmpparms
/var/log/secure
/var/log/messages
/var/log/audit/audit.log
/etc/ssh/sshd_config
/etc/nsswitch.conf
/etc/os-release
/etc/ssh/SshdRunningConfig
/etc/pam.d/password-auth
/etc/pam.d/system-auth
Q:
How to collect logs to troubleshoot PSMP issues?
A:
Note: Before restarting SSHD daemon, please make sure you have console access to PSMP server, in case SSHD daemon cannot start due to configuration issues.
In order to have complete logs we should enable the debug log level on the PSMP server:
1. From the PVWA UI, select "Administration", click on "Options", choose "Privileged Session Management" and in "General Settings" set parameters as of the following:
* ConnectionClient Settings -> TraceLevels=1,2
2. Please raise the debug level on the PSMP -
2.1 At command line, specify the following command to open the sshd file - vi /etc/ssh/sshd
2.2 At the beginning of the file, add the following lines:
export PSMPOpenSSHTraceLevels=1,2
export PSMPOpenSSHLogFolder=/var/opt/CARKpsmp/logs/components
We also need /var/log/secure file, to enable secure.log file:
1. vi /etc/ssh/sshd_config
2. Add LogLevel DEBUG3
3. Restart the sshd service to apply the changes:
/etc/init.d/sshd restart
4. Afterwards, please reproduce the problem and provide us with all of the below mentioned files, stating the reproduction time, user and target server name for further investigation.
5. Please provide the PSMP logs ( /var/opt/CARKpsmp/logs and /var/opt/CARKpsmp/logs/components and SSHD.log (/var/opt/CARKpsmp/logs/)), as well as the sshd_config, sshd, and the secure.log files.
At the end, you should have all of the following:
-SSHD_config file
-SSHD file
-PSMPConsole.log
-PSMPTrace.log
-The SSHD components logs from the components folder
-/var/log/secure.log (there may be multiple)
-/var/log/messages.log
-A screenshot of the exact error text you are seeing on the account (this may provide more information as to the cause)
Breakdown of TraceLevels:
0 – None.
1 – Exceptions only. Each error in the system will be sent to the trace file, whether it is recoverable or not.
2 – Controller trace. Includes the initialization of the PSMServer, recovery procedure and configuration.
3 – Session trace. Includes all the work done for a session (authentication and impersonation, password retrieval, activation of components, etc.).
4 – CASOS errors trace (Vault errors trace).
5 – CASOS debug and activity trace.