- Access exclusive content
- Connect with peers
- Share your expertise
- Find support resources
Our community just got stronger! Explore new identity security resources, connect with experts, and discover more ways to learn and collaborate.
01-23-2024 01:56 AM
Hi, we are facing vault switch over fail from node A to Node B & below are the error logs from CVM & due to switch over fail on primary vault , failover is getting initiate on DR site.
21/01/2024 23:59:45 [INFO] CVMCS148I Switchover command was issued from CyberArk Cluster Vault Management utility
21/01/2024 23:59:45 [INFO] CVMCS098I Shutting down resources.
22/01/2024 00:00:51 [ERROR] CVMSM079E Error stopping service PrivateArk Server. Reason: CVMSM081E Timeout has occurred. Code: 0.
22/01/2024 00:00:51 [ERROR] CVMCS100E Resources shutdown failed. Starting retry number 1.
22/01/2024 00:02:01 [ERROR] CVMSM079E Error stopping service PrivateArk Server. Reason: CVMSM081E Timeout has occurred. Code: 0.
22/01/2024 00:02:01 [ERROR] CVMCS101E Resources shutdown failed. Fix the issue and restart the Cluster Vault Manager service.
22/01/2024 00:07:03 [INFO] CVMCS098I Shutting down resources.
01-23-2024 03:09 AM
Hi @Saishiva Gandey
Can you please check the following?
Hope these steps helps !
01-23-2024 03:09 AM
Hi @Saishiva Gandey
Can you please check the following?
Hope these steps helps !
01-23-2024 03:58 AM
Node B isn't taking over? Instead the DR site is seeing the vault down and taking over?
This sort of error has numerous causes. Could be as simple as needing a reboot or could be complex eg wrong type of Quorum drive.
Apart from the above, any other clues in \PrivateArk\Server\ClusterVault\Logs\ClusterVault.console.log ?
01-23-2024 05:02 AM
@Malcolm Murray yes Node B isn't taking over is not taking over & instead the DR site is seeing the vault down and taking over for failover & yes your are correct when we restart the server private IP is pingable but every time we can't restart the server
01-23-2024 05:06 AM
@Malhar Vora Thanks for the response.
Check you can ping the Node B using its private IP address > yes, but during the switch over connection is breaking & when we restart the server again connection is getting up, but restart the which can't do everytime
Try disabling and enabling the Heartbeat NIC > should i do this on Node A & Node B on both nodes?
Stop all services using the CVM on Node A and Node B and then try to start all services using CVM from Node B.
Perform the switch over again from Node B to A and Node A to B.
Verify the SAN disks (data and quoroum) are in healthy status > data and quoroum disks are healthy
01-23-2024 06:39 AM
Hi @Saishiva Gandey
I was have a similar issue in one of the deployments.
Shutdown all cark services on both nodes, then turn offline all disks on all nodes.
Then using CVM start CARK on first node.
After first node getting up initiate moving from one node to secondary.
Seems that problem is NTFS issue caused quorum disk.
01-23-2024 07:09 AM
has this functioned previously or are you standing up a new HA environment and running into issues?
-Matt
01-23-2024 08:14 AM
We just experienced this same issue after upgrading our HA vault from 12.2 to 12.6.10. We see in ITALOG.log there is 1 active task that the active node is waiting on, but it never ends. This leads to the "CVMSM079E Error stopping service PrivateArk Server. Reason: CVMSM081E Timeout has occurred. Code: 0." error in ClusterVaultConsole.log.
The PrivateArk Server service gets stuck in a Stopping state. We can get it stopped by killing the dbmain.exe process, and then we can initiate a stop of the active node in CVM. Once the active node finally shuts down, the passive node will see this and start up and assume the active role. If we don't perform these steps, then our primary candidate DR vault would eventually initiate its failover process.
We have a case open with CyberArk at the moment.
01-23-2024 07:24 PM
Thanks for your replay , but our DR region is working fine, we did switch over multiple times on DR site we didn't face any issue
01-23-2024 07:25 PM
we recently implemented HA cluster, from the beginning we are seeing this issue but not in DR region
01-24-2024 03:13 AM
Hi @Saishiva Gandey
Yes please try disabling and enabling Heartbeat NIC on Primary Node B once and then try to switchover the nodes.
Hope it helps !
01-24-2024 05:20 AM
ok, so it's most likely something done incorrectly during setup and install. check your logs and review your setup settings.
-Matt
01-28-2024 12:36 AM
Hi @Malhar Vora we tried this & it worked 🙂 Thanks for the help , I have one query, first we did disable & enable the private hardening NIC on Node B & switched from node A to B and it was success but while switching back from node B to A we didn't disable & enable private NIC on Node A but still it was success without any issue, so should i do the same on Node A as well? However, it was success now but just wanted to check with you
Also should we do the same process every time whenever we need to initiate switch over activity ? can you pls confirm these 2 queries ...
01-28-2024 07:32 PM
Hi @Saishiva Gandey No, its not required on Node A as it is working fine. Please mark the above answer as 'Best Answer' if it has helped you. Thanks !
02-02-2024 09:36 AM
Our DR vault was working fine too, we only experienced this issue within our primary HA cluster.
The cause ended up being something simple; an LDAP sync was running at the time and it was causing the timeout during the switchover process. We later tried another switchover when LDAP sync wasn't running and it worked fine.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!

