Apply OS Updates to DAP Auto-Failover Nodes

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements

Welcome Idira to LIVEcommunity!

Our community just got stronger! Explore new identity security resources, connect with experts, and discover more ways to learn and collaborate.

Meet Idira

Secrets Management Articles
3 min read
L4 Transporter
No ratings

Introduction

This how-to guide describes how to apply patches, updates, and perform other routine maintenance on a DAP node in an auto-failover cluster without inadvertently triggering a failover.


Background

The DAP auto-failover feature detects when the active DAP Master becomes unresponsive and automatically promotes a DAP standby to take its place. However, the conditions caused by failure events (e.g. power outage, network loss, etc.) may also be caused by routine maintenance events that reboot nodes or cause the network to be temporarily unavailable.

When these routine maintenance events trigger a failover, it requires additional effort to repair the DAP cluster and return it to a state that is resilient to true failures again.


Alternative

This guide provides a procedure for temporarily disabling auto-failover during maintenance periods. However, many customers have successfully deployed auto-failover clusters with a longer time-to-live (TTL) intended to withstand these short-lived outages. For example, a TTL of 5 minutes (300 seconds) is often sufficient to allow the nodes to reboot without falsely triggering a failover.

See Modify Auto-Failover Cluster TTL  for more details on how to accomplish this.
 

Steps

Procedure for Patching an Auto-failover DAP Cluster

1. Disable cluster Service on Standbys

Disabling the cluster service on the Standbys prevents them from attempting to promote to Master while the active Master is temporarily down for maintenance.

For each Standby in the auto-failover cluster:

  1. Connect to the Standby node (e.g. using SSH)

  2. Disable the cluster service with the following command in the DAP container:

    sv down cluster
    For example:
    $ docker exec dap sv down cluster
    

It is not necessary to disable the cluster service on the active Master node.


2. Perform Node Maintenance

Once the cluster service is disabled on all Standbys, each of the nodes in the Master cluster may be brought offline, rebooted, etc. without triggering a failover from the active Master to a Standby.


3. Verify Master is Healthy

Once maintenance is complete, verify the Master is currently healthy according to its /health API endpoint:

...
"cluster": {
    "ok": true,
    "status": "running",
    "message": "acting as master"
  },
  "ok": true,
  "role": "master"


4. Re-enable the cluster Service on each Standby

For each Standby in the auto-failover cluster:

  1. Connect to the Standby node (e.g. using SSH)

  2. Re-enable the cluster service with the following command in the DAP container:

    sv up cluster
    

    For example:

    $ docker exec dap sv up cluster
    


5. Verify Auto-failover Cluster is Healthy

Once the cluster service is re-enabled on the Standbys, verify the cluster is healthy.

Verify the Master is currently healthy according to its /health API endpoint:

...
"cluster": {
    "ok": true,
    "status": "running",
    "message": "acting as master"
  },
  "ok": true,
  "role": "master"

Verify each Standby is healthy according to its /health API endpoint:

...
"cluster": {
    "ok": true,
    "status": "standing_by",
    "message": null
  },
  "ok": true,
  "role": "standby"
Rate this article:
  • 985 Views
  • 0 comments
  • 0 Likes
Article Dashboard
Version history
Last Updated:
‎04-21-2023 08:28 AM
Updated by: