Understanding the Disk and Database Corruption Issues that affect the Idira HA-Vault

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
PAM Self-Hosted Articles
5 min read
L1 Bithead
No ratings

Currently, there is a known issue that can result in the corruption of a vault database on an HA vault or possibly a stand-alone vault with attached storage.
Terms and architecture

The HA vault is designed to recover from issues like service crashes and hardware failures by providing two active physical servers. If a failure occurs on one server the other server should take over.
To achieve HA functionality some parts of the vault have added complexities to allow the active service to jump between both physical servers. The two major additions are the Cluster Vault Manager and network-attached storage.
The Cluster Vault Manager or CVM is used to orchestrate which resources are active and monitor for failures that would require the active service to be moved to the inactive server.
Shared storage allows a network disk to be presented to both servers. the vaults database and safe file system are installed on the shared storage.
Disk corruption is when writes to the disk are not committed due to the state of the disk
Database corruption is when the MySQL server service will no longer start because of filesystem-level failures in the database.

High-level issue description

The database corruption issue the majority of the time happens in two parts.
  1. The disk becomes corrupted
  2. The database attempts to write to the corrupted disk resulting in lost data
  3. The database will either crash or no longer be able to start.

Steps

Corrupting the disk

How the disk becomes corrupted is not fully documented but in the majority of documented cases, the issue stems from trying to bring the shared storage online on both servers.
This issue is advised against in the Vault install documentation and the CVM is designed to not allow this to happen.
From previous cases, the most common reason the storage is brought online on both servers is not relying on the CVM to manage resources and starting resources manually. There are a handful of use cases where the CVM needs to be bypassed, one is the initial configuration of the HA vault where the CVM has not been configured and is in an upgrade.
Overall, the CVM should always be used to manage resources. If the CVM is not starting resources or the HA vault in general it most likely is finding a failure and refusing to continue. A known issue is that the CVM GUI does not show these errors and the admin will need to reference the CVM console and trace on both servers to understand why the CVM is not starting the vault.
Many database corruption cases start here.
 

Corrupting the database

When the CVM is refusing to start resources in these database corruption issues the admins will attempt to bypass the CVM.
One of the reasons the CVM will refuse to start is that it can detect if a disk is corrupt:
ka3Vy000009IFy0IAG__eid=ka3Vy000009IFy0&feoid=00N2J000009R3kv&refid=0EM2J000002iR5i.png
When the CVM sees the disk is corrupt it will not allow the other services to start.
This is where bad practice corrupts the database. Admins will bring the shared storage online and then attempts to start the vault services starting with the database. None of these services are designed to detect disk corruption like the CVM is.
There are a few actions that take place at this point.
  1. The database service starts but is running on a corrupted disk
  2. The database detects corruption and attempts to recover using internal MySQL functions, this then becomes #1
  3. The database just does not start.

Understanding the corrupted database

The MySQL database is designed to recover from crashes and other minor events where it is not cleanly shutdown:
ka3Vy000009IFy0IAG__eid=ka3Vy000009IFy0&feoid=00N2J000009R3kv&refid=0EM2J000002iR5j.png
The bad news for many customers is that their bad practices of bypassing the CVM will usually push them past the point MySQL will be able to self-recover.
In the case, this log screenshot is from the customer did not detect the corrupted disk and most likely did not realize the database recovered itself. At this point, the recovery actually happened on a corrupt disk meaning it did not actually recover. Furthermore, in this case, and many others the customer tried to start the database service many times further compacting the issue.
 

Recovering from a corrupted database or disk.

Recovering the disk and or database is not a guaranteed process and a rebuild of the database of the vault may still be needed. From the best practice perspective, the following process should prevent and recover from disk corruption.
  1. Always rely on the CVM for resource management unless working with Idira technical resources
  2. If the CVM is refusing to start the vault on one of the HA servers or all together review the CVM console and trace logs for the reason why
  3. If the disk is marked as dirty use chkdsk to attempt to fix the disk
  4. Review the vaultdb.log for issues with MySQL
  5. DO NOT ATTEMPT TO START THE VAULT DATABASE SERVICE REPEATEDLY IF IT IS CRASHING OR STOPPING ITSELF
Rate this article: