<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" version="2.0">
  <channel>
    <title>article Understanding the Disk and Database Corruption Issues that affect the Idira HA-Vault in PAM Self-Hosted Articles</title>
    <link>https://live.paloaltonetworks.com/t5/pam-self-hosted-articles/understanding-the-disk-and-database-corruption-issues-that/ta-p/1427970</link>
    <description>&lt;H1 style="margin-top: 16px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Currently, there is a known issue that can result in the corruption of a vault database on an HA vault or possibly a stand-alone vault with attached storage.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 16pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: &amp;quot;Calibri Light&amp;quot;,sans-serif;"&gt;&lt;SPAN style="color: #2f5496;"&gt;&lt;SPAN style="font-weight: normal;"&gt;Terms and architecture&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/H1&gt;
&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The HA vault is designed to recover from issues like service crashes and hardware failures by providing two active physical servers. If a failure occurs on one server the other server should take over.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;To achieve HA functionality some parts of the vault have added complexities to allow the active service to jump between both physical servers. The two major additions are the Cluster Vault Manager and network-attached storage.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The Cluster Vault Manager or CVM is used to orchestrate which resources are active and monitor for failures that would require the active service to be moved to the inactive server.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Shared storage allows a network disk to be presented to both servers. the vaults database and safe file system are installed on the shared storage.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Disk corruption is when writes to the disk are not committed due to the state of the disk&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Database corruption is when the MySQL server service will no longer start because of filesystem-level failures in the database.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;
&lt;H1 style="margin-top: 16px;"&gt;&lt;SPAN style="font-size: 16pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: &amp;quot;Calibri Light&amp;quot;,sans-serif;"&gt;&lt;SPAN style="color: #2f5496;"&gt;&lt;SPAN style="font-weight: normal;"&gt;High-level issue description&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/H1&gt;
&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The database corruption issue the majority of the time happens in two parts.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;
&lt;OL&gt;&lt;LI style="margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The disk becomes corrupted&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI style="margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The database attempts to write to the corrupted disk resulting in lost data&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI style="margin-bottom: 11px;margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The database will either crash or no longer be able to start.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;/OL&gt;
&lt;H3&gt;Steps&lt;/H3&gt;
&lt;H2 style="margin-top: 3px;"&gt;&lt;SPAN style="font-size: 13pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: &amp;quot;Calibri Light&amp;quot;,sans-serif;"&gt;&lt;SPAN style="color: #2f5496;"&gt;&lt;SPAN style="font-weight: normal;"&gt;Corrupting the disk&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/H2&gt;
&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;How the disk becomes corrupted is not fully documented but in the majority of documented cases, the issue stems from trying to bring the shared storage online on both servers.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;This issue is advised against in the Vault install documentation and the CVM is designed to not allow this to happen.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;From previous cases, the most common reason the storage is brought online on both servers is not relying on the CVM to manage resources and starting resources manually. There are a handful of use cases where the CVM needs to be bypassed, one is the initial configuration of the HA vault where the CVM has not been configured and is in an upgrade.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Overall, the CVM should always be used to manage resources. If the CVM is not starting resources or the HA vault in general it most likely is finding a failure and refusing to continue. A known issue is that the CVM GUI does not show these errors and the admin will need to reference the CVM console and trace on both servers to understand why the CVM is not starting the vault.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Many database corruption cases start here.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&amp;nbsp;
&lt;H2 style="margin-top: 3px;"&gt;&lt;SPAN style="font-size: 13pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: &amp;quot;Calibri Light&amp;quot;,sans-serif;"&gt;&lt;SPAN style="color: #2f5496;"&gt;&lt;SPAN style="font-weight: normal;"&gt;Corrupting the database&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/H2&gt;
&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;When the CVM is refusing to start resources in these database corruption issues the admins will attempt to bypass the CVM.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;One of the reasons the CVM will refuse to start is that it can detect if a disk is corrupt:&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;&lt;span class="lia-inline-image-display-wrapper" image-alt="ka3Vy000009IFy0IAG__eid=ka3Vy000009IFy0&amp;amp;feoid=00N2J000009R3kv&amp;amp;refid=0EM2J000002iR5i.png"&gt;&lt;img src="https://live.paloaltonetworks.com/t5/image/serverpage/image-id/96778i6F428994D183A6A7/image-size/large?v=v2&amp;amp;px=999" role="button" title="ka3Vy000009IFy0IAG__eid=ka3Vy000009IFy0&amp;amp;feoid=00N2J000009R3kv&amp;amp;refid=0EM2J000002iR5i.png" alt="ka3Vy000009IFy0IAG__eid=ka3Vy000009IFy0&amp;amp;feoid=00N2J000009R3kv&amp;amp;refid=0EM2J000002iR5i.png" /&gt;&lt;/span&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;When the CVM sees the disk is corrupt it will not allow the other services to start.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;This is where bad practice corrupts the database. Admins will bring the shared storage online and then attempts to start the vault services starting with the database. None of these services are designed to detect disk corruption like the CVM is.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;There are a few actions that take place at this point.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;
&lt;OL&gt;&lt;LI style="margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The database service starts but is running on a corrupted disk&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI style="margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The database detects corruption and attempts to recover using internal MySQL functions, this then becomes #1&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI style="margin-bottom: 11px;margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The database just does not start.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;/OL&gt;
&lt;H1 style="margin-top: 16px;"&gt;&lt;SPAN style="font-size: 16pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: &amp;quot;Calibri Light&amp;quot;,sans-serif;"&gt;&lt;SPAN style="color: #2f5496;"&gt;&lt;SPAN style="font-weight: normal;"&gt;Understanding the corrupted database&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/H1&gt;
&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The MySQL database is designed to recover from crashes and other minor events where it is not cleanly shutdown:&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;&lt;span class="lia-inline-image-display-wrapper" image-alt="ka3Vy000009IFy0IAG__eid=ka3Vy000009IFy0&amp;amp;feoid=00N2J000009R3kv&amp;amp;refid=0EM2J000002iR5j.png"&gt;&lt;img src="https://live.paloaltonetworks.com/t5/image/serverpage/image-id/96779i8C3A85A13751B152/image-size/large?v=v2&amp;amp;px=999" role="button" title="ka3Vy000009IFy0IAG__eid=ka3Vy000009IFy0&amp;amp;feoid=00N2J000009R3kv&amp;amp;refid=0EM2J000002iR5j.png" alt="ka3Vy000009IFy0IAG__eid=ka3Vy000009IFy0&amp;amp;feoid=00N2J000009R3kv&amp;amp;refid=0EM2J000002iR5j.png" /&gt;&lt;/span&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The bad news for many customers is that their bad practices of bypassing the CVM will usually push them past the point MySQL will be able to self-recover.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;In the case, this log screenshot is from the customer did not detect the corrupted disk and most likely did not realize the database recovered itself. At this point, the recovery actually happened on a corrupt disk meaning it did not actually recover. Furthermore, in this case, and many others the customer tried to start the database service many times further compacting the issue.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&amp;nbsp;
&lt;H1 style="margin-top: 16px;"&gt;&lt;SPAN style="font-size: 16pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: &amp;quot;Calibri Light&amp;quot;,sans-serif;"&gt;&lt;SPAN style="color: #2f5496;"&gt;&lt;SPAN style="font-weight: normal;"&gt;Recovering from a corrupted database or disk.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/H1&gt;
&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Recovering the disk and or database is not a guaranteed process and a rebuild of the database of the vault may still be needed. From the best practice perspective, the following process should prevent and recover from disk corruption.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;
&lt;OL&gt;&lt;LI style="margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Always rely on the CVM for resource management unless working with Idira technical resources&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI style="margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;If the CVM is refusing to start the vault on one of the HA servers or all together review the CVM console and trace logs for the reason why&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI style="margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;If the disk is marked as dirty use chkdsk to attempt to fix the disk&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI style="margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Review the vaultdb.log for issues with MySQL&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI style="margin-bottom: 11px;margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;&lt;SPAN style="color: red;"&gt;DO NOT ATTEMPT TO START THE VAULT DATABASE SERVICE REPEATEDLY IF IT IS CRASHING OR STOPPING ITSELF&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;/OL&gt;</description>
    <pubDate>Thu, 05 Nov 2026 20:07:07 GMT</pubDate>
    <dc:creator>CyberArkAlumni</dc:creator>
    <dc:date>2026-11-05T20:07:07Z</dc:date>
    <item>
      <title>Understanding the Disk and Database Corruption Issues that affect the Idira HA-Vault</title>
      <link>https://live.paloaltonetworks.com/t5/pam-self-hosted-articles/understanding-the-disk-and-database-corruption-issues-that/ta-p/1427970</link>
      <description>&lt;H1 style="margin-top: 16px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Currently, there is a known issue that can result in the corruption of a vault database on an HA vault or possibly a stand-alone vault with attached storage.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 16pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: &amp;quot;Calibri Light&amp;quot;,sans-serif;"&gt;&lt;SPAN style="color: #2f5496;"&gt;&lt;SPAN style="font-weight: normal;"&gt;Terms and architecture&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/H1&gt;
&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The HA vault is designed to recover from issues like service crashes and hardware failures by providing two active physical servers. If a failure occurs on one server the other server should take over.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;To achieve HA functionality some parts of the vault have added complexities to allow the active service to jump between both physical servers. The two major additions are the Cluster Vault Manager and network-attached storage.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The Cluster Vault Manager or CVM is used to orchestrate which resources are active and monitor for failures that would require the active service to be moved to the inactive server.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Shared storage allows a network disk to be presented to both servers. the vaults database and safe file system are installed on the shared storage.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Disk corruption is when writes to the disk are not committed due to the state of the disk&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Database corruption is when the MySQL server service will no longer start because of filesystem-level failures in the database.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;
&lt;H1 style="margin-top: 16px;"&gt;&lt;SPAN style="font-size: 16pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: &amp;quot;Calibri Light&amp;quot;,sans-serif;"&gt;&lt;SPAN style="color: #2f5496;"&gt;&lt;SPAN style="font-weight: normal;"&gt;High-level issue description&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/H1&gt;
&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The database corruption issue the majority of the time happens in two parts.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;
&lt;OL&gt;&lt;LI style="margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The disk becomes corrupted&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI style="margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The database attempts to write to the corrupted disk resulting in lost data&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI style="margin-bottom: 11px;margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The database will either crash or no longer be able to start.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;/OL&gt;
&lt;H3&gt;Steps&lt;/H3&gt;
&lt;H2 style="margin-top: 3px;"&gt;&lt;SPAN style="font-size: 13pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: &amp;quot;Calibri Light&amp;quot;,sans-serif;"&gt;&lt;SPAN style="color: #2f5496;"&gt;&lt;SPAN style="font-weight: normal;"&gt;Corrupting the disk&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/H2&gt;
&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;How the disk becomes corrupted is not fully documented but in the majority of documented cases, the issue stems from trying to bring the shared storage online on both servers.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;This issue is advised against in the Vault install documentation and the CVM is designed to not allow this to happen.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;From previous cases, the most common reason the storage is brought online on both servers is not relying on the CVM to manage resources and starting resources manually. There are a handful of use cases where the CVM needs to be bypassed, one is the initial configuration of the HA vault where the CVM has not been configured and is in an upgrade.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Overall, the CVM should always be used to manage resources. If the CVM is not starting resources or the HA vault in general it most likely is finding a failure and refusing to continue. A known issue is that the CVM GUI does not show these errors and the admin will need to reference the CVM console and trace on both servers to understand why the CVM is not starting the vault.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Many database corruption cases start here.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&amp;nbsp;
&lt;H2 style="margin-top: 3px;"&gt;&lt;SPAN style="font-size: 13pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: &amp;quot;Calibri Light&amp;quot;,sans-serif;"&gt;&lt;SPAN style="color: #2f5496;"&gt;&lt;SPAN style="font-weight: normal;"&gt;Corrupting the database&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/H2&gt;
&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;When the CVM is refusing to start resources in these database corruption issues the admins will attempt to bypass the CVM.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;One of the reasons the CVM will refuse to start is that it can detect if a disk is corrupt:&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;&lt;span class="lia-inline-image-display-wrapper" image-alt="ka3Vy000009IFy0IAG__eid=ka3Vy000009IFy0&amp;amp;feoid=00N2J000009R3kv&amp;amp;refid=0EM2J000002iR5i.png"&gt;&lt;img src="https://live.paloaltonetworks.com/t5/image/serverpage/image-id/96778i6F428994D183A6A7/image-size/large?v=v2&amp;amp;px=999" role="button" title="ka3Vy000009IFy0IAG__eid=ka3Vy000009IFy0&amp;amp;feoid=00N2J000009R3kv&amp;amp;refid=0EM2J000002iR5i.png" alt="ka3Vy000009IFy0IAG__eid=ka3Vy000009IFy0&amp;amp;feoid=00N2J000009R3kv&amp;amp;refid=0EM2J000002iR5i.png" /&gt;&lt;/span&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;When the CVM sees the disk is corrupt it will not allow the other services to start.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;This is where bad practice corrupts the database. Admins will bring the shared storage online and then attempts to start the vault services starting with the database. None of these services are designed to detect disk corruption like the CVM is.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;There are a few actions that take place at this point.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;
&lt;OL&gt;&lt;LI style="margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The database service starts but is running on a corrupted disk&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI style="margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The database detects corruption and attempts to recover using internal MySQL functions, this then becomes #1&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI style="margin-bottom: 11px;margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The database just does not start.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;/OL&gt;
&lt;H1 style="margin-top: 16px;"&gt;&lt;SPAN style="font-size: 16pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: &amp;quot;Calibri Light&amp;quot;,sans-serif;"&gt;&lt;SPAN style="color: #2f5496;"&gt;&lt;SPAN style="font-weight: normal;"&gt;Understanding the corrupted database&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/H1&gt;
&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The MySQL database is designed to recover from crashes and other minor events where it is not cleanly shutdown:&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;&lt;span class="lia-inline-image-display-wrapper" image-alt="ka3Vy000009IFy0IAG__eid=ka3Vy000009IFy0&amp;amp;feoid=00N2J000009R3kv&amp;amp;refid=0EM2J000002iR5j.png"&gt;&lt;img src="https://live.paloaltonetworks.com/t5/image/serverpage/image-id/96779i8C3A85A13751B152/image-size/large?v=v2&amp;amp;px=999" role="button" title="ka3Vy000009IFy0IAG__eid=ka3Vy000009IFy0&amp;amp;feoid=00N2J000009R3kv&amp;amp;refid=0EM2J000002iR5j.png" alt="ka3Vy000009IFy0IAG__eid=ka3Vy000009IFy0&amp;amp;feoid=00N2J000009R3kv&amp;amp;refid=0EM2J000002iR5j.png" /&gt;&lt;/span&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;The bad news for many customers is that their bad practices of bypassing the CVM will usually push them past the point MySQL will be able to self-recover.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;In the case, this log screenshot is from the customer did not detect the corrupted disk and most likely did not realize the database recovered itself. At this point, the recovery actually happened on a corrupt disk meaning it did not actually recover. Furthermore, in this case, and many others the customer tried to start the database service many times further compacting the issue.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;BR /&gt;&amp;nbsp;
&lt;H1 style="margin-top: 16px;"&gt;&lt;SPAN style="font-size: 16pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: &amp;quot;Calibri Light&amp;quot;,sans-serif;"&gt;&lt;SPAN style="color: #2f5496;"&gt;&lt;SPAN style="font-weight: normal;"&gt;Recovering from a corrupted database or disk.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/H1&gt;
&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Recovering the disk and or database is not a guaranteed process and a rebuild of the database of the vault may still be needed. From the best practice perspective, the following process should prevent and recover from disk corruption.&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;
&lt;OL&gt;&lt;LI style="margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Always rely on the CVM for resource management unless working with Idira technical resources&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI style="margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;If the CVM is refusing to start the vault on one of the HA servers or all together review the CVM console and trace logs for the reason why&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI style="margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;If the disk is marked as dirty use chkdsk to attempt to fix the disk&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI style="margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;Review the vaultdb.log for issues with MySQL&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;LI style="margin-bottom: 11px;margin-left: 8px;"&gt;&lt;SPAN style="font-size: 11pt;"&gt;&lt;SPAN style="line-height: 107%;"&gt;&lt;SPAN style="font-family: Calibri,sans-serif;"&gt;&lt;SPAN style="color: red;"&gt;DO NOT ATTEMPT TO START THE VAULT DATABASE SERVICE REPEATEDLY IF IT IS CRASHING OR STOPPING ITSELF&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/SPAN&gt;&lt;/LI&gt;&lt;/OL&gt;</description>
      <pubDate>Thu, 05 Nov 2026 20:07:07 GMT</pubDate>
      <guid>https://live.paloaltonetworks.com/t5/pam-self-hosted-articles/understanding-the-disk-and-database-corruption-issues-that/ta-p/1427970</guid>
      <dc:creator>CyberArkAlumni</dc:creator>
      <dc:date>2026-11-05T20:07:07Z</dc:date>
    </item>
  </channel>
</rss>

