DRVault Sync error; Any Ideas?

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements

Welcome Idira to LIVEcommunity!

Our community just got stronger! Explore new identity security resources, connect with experts, and discover more ways to learn and collaborate.

Meet Idira

DRVault Sync error; Any Ideas?

L2 Linker

Hi,

 

we have a Vault an three DRVaults .

 

One of this DRVault has since weeks problems.. After a switch from DRVault to Vault and back to DRVault, the sync has this kind of errors....

 

[24/09/2019  10:12:51.360071]  ::  PADR0009I Replicate started. 

[24/09/2019  10:12:51.360091]  ::  PADR0069I Resetting metadata backup environment. 

[24/09/2019  10:12:51.360099]  ::  PADR0065I Replicating Backup key to D:\PAKeys\Backup.key. 

[24/09/2019  10:12:51.407690]  ::  PADR0095I Refreshing Vault configuration files. 

[24/09/2019  10:12:51.624219]  ::  PADR0097I Refreshing Vault configuration files completed successfully. 

[24/09/2019  10:12:51.624352]  ::  PADR0089I Stopping Slave. 

[24/09/2019  10:12:51.624588]  ::  PADR0088I Triggering Full Replication. 

[24/09/2019  10:12:51.624642]  ::  PAREP061I Generating Full backup. 

[24/09/2019  12:03:08.682742]  ::  CASTM003E Vault transaction failed. Reason: ITAPE266E Send object failed (code = 3, 0) . ITAPE266E Send object failed (code = 3, 0) 

[24/09/2019  12:03:08.682792]  ::  PADR0005E Full Replicate failed. (Code -1409266) 

[24/09/2019  12:03:08.828064]  ::  PADR0005E Reset Slave failed (Code -1409266) 

[24/09/2019  12:03:08.828105]  ::  PADR0011E Replicate failed. 

[24/09/2019  12:03:09.939072]  ::  PADR0100W Metadata Replication warning: Communication error. See Diagnostic Info below. 

[24/09/2019  12:03:09.939111]  ::   Last SQL Error Code: 0 

[24/09/2019  12:03:09.939132]  ::   Last SQL Error: "" 

[24/09/2019  12:03:09.939150]  ::   Last IO Error Code: 0 

[24/09/2019  12:03:09.939167]  ::   Last IO Error: "" 

[24/09/2019  12:03:09.939185]  ::   SQL Thread Running State: "No" 

[24/09/2019  12:03:09.939204]  ::   IO Thread Running State: "No" 

[24/09/2019  12:08:08.707039]  ::  PADR0009I Replicate started. 

[24/09/2019  12:08:09.193847]  ::  PAREP013I Replicating Safe System. 

[24/09/2019  12:08:18.623334]  ::  PAREP013I Replicating Safe Pictures. 

[24/09/2019  12:08:18.641404]  ::  PAREP013I Replicating Safe VaultInternal. 

 

We have done a lot of thinks, but nothing works... Also from the Support in the moment not really an idea...

 

What we have done:

  • Uninstall DRVault component and install again
  • Uninstall, Vault and DRVault and install again
  • Uninstall Vault and DRVault and also deleted the DR User, all installed again
  • Uninstall Vault and DRVault, deleted all File from the Server (in Server, PADR and PrivateArk), install again
  • Uninstall Vault and DRVault, deleted all File from the Server (in Server, PADR and PrivateArk), install again with an different DR User

 

Every time after the Sync start we have the same error.

 

A switch from DRVault to Vault didn't work.

 

Any Ideas?

 

Kind regards

Francesco

 

14 REPLIES 14

L2 Linker

And it works before we have done the Upgrade to 10.7 and 10.10!

 

On May we have done a Disaster Recovery Test and this DRVault was working!

 

The other two DRVault have not this error.

L4 Transporter

Have you tried formatting the drive where the Safes and Metadata resides? and then performing a Full Replication?

 

L2 Linker

no, a formatting from the drive was not done. But everytime a Fullreplication.

L4 Transporter

I had similar errors in our DR replication logs along with the errors on event viewer pointing to the NTFS corruption.

I formatted the disk where the data and metadata resides (in our case it is SAN), rebuilt the DR and ran a chkdsk /F on the disk. This fixed all the issues.

You probably don't need to run chkdsk if there are no ntfs errors but you can try formatting.

 

Also, please check the time sync between this and Primary Vault.

L2 Linker

OK, I've started to formatting the disk.... Tomorrow I start the installation.

L2 Linker

So... Install done, Resync started => Same error:

 

[25/09/2019  11:34:52.738714]  ::  PADR0097I Refreshing Vault configuration files completed successfully. 

[25/09/2019  11:34:52.738852]  ::  PADR0089I Stopping Slave. 

[25/09/2019  11:34:52.739014]  ::  PADR0088I Triggering Full Replication. 

[25/09/2019  11:34:52.739034]  ::  PAREP061I Generating Full backup. 

[25/09/2019  13:24:37.794326]  ::  CASTM003E Vault transaction failed. Reason: ITAPE266E Send object failed (code = 3, 0) . ITAPE266E Send object failed (code = 3, 0) 

[25/09/2019  13:24:37.794351]  ::  PADR0005E Full Replicate failed. (Code -1409266) 

[25/09/2019  13:24:37.940783]  ::  PADR0005E Reset Slave failed (Code -1409266) 

[25/09/2019  13:24:37.940823]  ::  PADR0011E Replicate failed. 

[25/09/2019  13:24:39.025288]  ::  PADR0100W Metadata Replication warning: Communication error. See Diagnostic Info below. 

[25/09/2019  13:24:39.025329]  ::   Last SQL Error Code: 0 

[25/09/2019  13:24:39.025350]  ::   Last SQL Error: "" 

[25/09/2019  13:24:39.025368]  ::   Last IO Error Code: 0 

[25/09/2019  13:24:39.025386]  ::   Last IO Error: "" 

[25/09/2019  13:24:39.025404]  ::   SQL Thread Running State: "No" 

[25/09/2019  13:24:39.025422]  ::   IO Thread Running State: "No" 

[25/09/2019  13:29:37.310287]  ::  PADR0009I Replicate started. 

[25/09/2019  13:29:37.808272]  ::  PAREP013I Replicating Safe System. 

[25/09/2019  13:29:47.473506]  ::  PAREP013I Replicating Safe Pictures. 

[25/09/2019  13:29:47.484290]  ::  PAREP013I Replicating Safe VaultInternal. 

 

 

L4 Transporter

Can you check this article, if not already checked. Maybe its related?

https://community.cyberark.com/s/feed/0D52J00006RdxtiSAB

L4 Transporter

And are there any errors in your VaultDB.log related mysql-bin.xxxx?

L2 Linker

For CyberArk I've to install the complete OS also.... So reinstall started.

L2 Linker

I have same problem , I can increase prod and dr slave_net_timeout=30000 in (C:\Program Files (x86)\PrivateArk\Server\Database\my.ini ).

After this operation you can restart prod vault and dr service.

Thank you.

L2 Linker

OK, I will test it.... No solutions since yet.

L2 Linker

Sync done, same problem... The my.ini settings doesn't fit.

L0 Member

HI fcerasa, did you resole the issue?

L2 Linker

Hi slawomir.batog,

 

no problem is still existing... For a little timeframe the problem was solved and after a Vault activation from the Vault and turned back to DRVault, the problem is available again.

 

For me it's strange that we have this situation only on this DRVault. All other tweo DRVaults are working fine, also after a Vault activation.

  • 1498 Views
  • 14 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!