Vault Switch over getting failed from Node A to Node B

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements

Welcome Idira to LIVEcommunity!

Our community just got stronger! Explore new identity security resources, connect with experts, and discover more ways to learn and collaborate.

Meet Idira

Vault Switch over getting failed from Node A to Node B

L3 Networker

Hi, we are facing vault switch over fail from node A to Node B & below are the error logs from CVM & due to switch over fail on primary vault , failover is getting initiate on DR site.

 

21/01/2024 23:59:45 [INFO]  CVMCS148I Switchover command was issued from CyberArk Cluster Vault Management utility

21/01/2024 23:59:45 [INFO]  CVMCS098I Shutting down resources.

22/01/2024 00:00:51 [ERROR] CVMSM079E Error stopping service PrivateArk Server. Reason: CVMSM081E Timeout has occurred. Code: 0.

22/01/2024 00:00:51 [ERROR] CVMCS100E Resources shutdown failed. Starting retry number 1.

22/01/2024 00:02:01 [ERROR] CVMSM079E Error stopping service PrivateArk Server. Reason: CVMSM081E Timeout has occurred. Code: 0.

22/01/2024 00:02:01 [ERROR] CVMCS101E Resources shutdown failed. Fix the issue and restart the Cluster Vault Manager service.

22/01/2024 00:07:03 [INFO]  CVMCS098I Shutting down resources.

 

 

1 accepted solution

Accepted Solutions

L7 Applicator

Hi @Saishiva Gandey​ 

 

Can you please check the following?

 

  1. Check you can ping the Node B using its private IP address
  2. Try disabling and enabling the Heartbeat NIC
  3. Stop all services using the CVM on Node A and Node B and then try to start all services using CVM from Node B.
  4. Perform the switch over again from Node B to A and Node A to B.
  5. Verify the SAN disks (data and quoroum) are in healthy status

 

Hope these steps helps !

View solution in original post

18 REPLIES 18

L7 Applicator

Hi @Saishiva Gandey​ 

 

Can you please check the following?

 

  1. Check you can ping the Node B using its private IP address
  2. Try disabling and enabling the Heartbeat NIC
  3. Stop all services using the CVM on Node A and Node B and then try to start all services using CVM from Node B.
  4. Perform the switch over again from Node B to A and Node A to B.
  5. Verify the SAN disks (data and quoroum) are in healthy status

 

Hope these steps helps !

L7 Applicator

Node B isn't taking over? Instead the DR site is seeing the vault down and taking over?

 

This sort of error has numerous causes. Could be as simple as needing a reboot or could be complex eg wrong type of Quorum drive.

 

Apart from the above, any other clues in \PrivateArk\Server\ClusterVault\Logs\ClusterVault.console.log ?

@Malcolm Murray​  yes Node B isn't taking over is not taking over & instead the DR site is seeing the vault down and taking over for failover & yes your are correct when we restart the server private IP is pingable but every time we can't restart the server

@Malhar Vora​ Thanks for the response.

 

Check you can ping the Node B using its private IP address > yes, but during the switch over connection is breaking & when we restart the server again connection is getting up, but restart the which can't do everytime

Try disabling and enabling the Heartbeat NIC > should i do this on Node A & Node B on both nodes?

Stop all services using the CVM on Node A and Node B and then try to start all services using CVM from Node B.

Perform the switch over again from Node B to A and Node A to B.

Verify the SAN disks (data and quoroum) are in healthy status > data and quoroum disks are healthy

L2 Linker

Hi @Saishiva Gandey​ 

I was have a similar issue in one of the deployments.

Shutdown all cark services on both nodes, then turn offline all disks on all nodes.

Then using CVM start CARK on first node.

After first node getting up initiate moving from one node to secondary.

Seems that problem is NTFS issue caused quorum disk.

L7 Applicator

has this functioned previously or are you standing up a new HA environment and running into issues?

 

-Matt

L2 Linker

We just experienced this same issue after upgrading our HA vault from 12.2 to 12.6.10. We see in ITALOG.log there is 1 active task that the active node is waiting on, but it never ends. This leads to the "CVMSM079E Error stopping service PrivateArk Server. Reason: CVMSM081E Timeout has occurred. Code: 0." error in ClusterVaultConsole.log.

 

The PrivateArk Server service gets stuck in a Stopping state. We can get it stopped by killing the dbmain.exe process, and then we can initiate a stop of the active node in CVM. Once the active node finally shuts down, the passive node will see this and start up and assume the active role. If we don't perform these steps, then our primary candidate DR vault would eventually initiate its failover process.

 

We have a case open with CyberArk at the moment.

Thanks for your replay , but our DR region is working fine, we did switch over multiple times on DR site we didn't face any issue

we recently implemented HA cluster, from the beginning we are seeing this issue but not in DR region

Hi @Saishiva Gandey​ 

 

Yes please try disabling and enabling Heartbeat NIC on Primary Node B once and then try to switchover the nodes.

 

Hope it helps !

ok, so it's most likely something done incorrectly during setup and install. check your logs and review your setup settings.

 

-Matt

Hi @Malhar Vora​  we tried this & it worked 🙂 Thanks for the help , I have one query, first we did disable & enable the private hardening NIC on Node B & switched from node A to B and it was success but while switching back from node B to A we didn't disable & enable private NIC on Node A but still it was success without any issue, so should i do the same on Node A as well? However, it was success now but just wanted to check with you

 

Also should we do the same process every time whenever we need to initiate switch over activity ? can you pls confirm these 2 queries ...

Hi @Saishiva Gandey​ No, its not required on Node A as it is working fine. Please mark the above answer as 'Best Answer' if it has helped you. Thanks !

Our DR vault was working fine too, we only experienced this issue within our primary HA cluster.

 

The cause ended up being something simple; an LDAP sync was running at the time and it was causing the timeout during the switchover process. We later tried another switchover when LDAP sync wasn't running and it worked fine.

  • 1 accepted solution
  • 967 Views
  • 18 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!