- Access exclusive content
- Connect with peers
- Share your expertise
- Find support resources
Our community just got stronger! Explore new identity security resources, connect with experts, and discover more ways to learn and collaborate.
05-30-2024 12:42 AM
We are using Conjur Enterprises 13.2.2 with 1 leader and 2 standby nodes hosted on Azure VM’s.
We have uninstalled the openssh package from Standby node after which we were not able to ssh to Standby node. While investing we tried to restart the VM and later install the openssh package.
After VM restart conjur standby container didn’t started and we manually started it after few hours. After that when we checked the health api of standby node it was in initializing state and later found out conjur service was down. So manually started it. Still we are observing cluster status is in “initializing” state.
“cluster”: {
“ok”: true,
“degraded”: true,
“status”: “initializing”,
“message”: null
},
Log snapshot:
<134>1 2024-05-27T11:22:17.000+00:00 490a1925ee99 evoke-cluster - - [meta sequenceId=“756”] I, [2024-05-27T11:22:17.088894 #1659] INFO – : State is now initializing
<134>1 2024-05-27T11:22:35.000+00:00 490a1925ee99 etcd - - [meta sequenceId=“757”] {“level”:“warn”,“ts”:“2024-05-27T11:22:35.673391Z”,“caller”:“etcdserver/server.go:2083”,“msg”:“failed to publish local member to cluster through raft”,“local-member-id”:“643f457b26562629”,“local-member-attributes”:“{Name:x.x.x.c ClientURLs:[http://127.0.0.1:2379]}”,“request-path”:“/0/members/643f457b26562629/attributes”,“publish-timeout”:“25s”,“error”:“etcdserver: request timed out”}
<134>1 2024-05-27T11:23:00.000+00:00 490a1925ee99 etcd - - [meta sequenceId=“758”] {“level”:“warn”,“ts”:“2024-05-27T11:23:00.673968Z”,“caller”:“etcdserver/server.go:2083”,“msg”:“failed to publish local member to cluster through raft”,“local-member-id”:“643f457b26562629”,“local-member-attributes”:“{Name:x.x.x.x ClientURLs:[http://127.0.0.1:2379]}”,“request-path”:“/0/members/643f457b26562629/attributes”,“publish-timeout”:“25s”,“error”:“etcdserver: request timed out”}
<134>1 2024-05-27T11:23:25.000+00:00 490a1925ee99 etcd - - [meta sequenceId=“759”] {“level”:“warn”,“ts”:“2024-05-27T11:23:25.675059Z”,“caller”:“etcdserver/server.go:2083”,“msg”:“failed to publish local member to cluster through raft”,“local-member-id”:“643f457b26562629”,“local-member-attributes”:“{Name:x.x.x.x ClientURLs:[http://127.0.0.1:2379]}”,“request-path”:“/0/members/643f457b26562629/attributes”,“publish-timeout”:“25s”,“error”:“etcdserver: request timed out”}
06-02-2024 11:57 PM
Hi @Rupesh Sahu
It seems to me that the standby node is having issues with the etcd service.
Hope it helps !
06-04-2024 02:58 AM
Hi @Rupesh Sahu
Were you able to fix this issue?
Has restarting the 'etcd' service helped in this case or re-seeding from the leader & unpacking/configuring?
06-05-2024 02:06 AM
I dont see etcd service through systemctl status, so not able to start as well. But through pstree i can see etcd process is running.
─runsv─┬─logger
│ └─start_etcd─┬─etcd───6*[{etcd}]
│ ├─grep
│ └─tee
06-05-2024 02:10 AM
Not able to see etcd service.
I can do re-seeding but currently not sure whether some issue with leader as well.
we are observing below issue in our setup:
1. As per the leader health check, there is no issue in leader node
2. As per the standby2 health check, node is initializing
3. There is no replication happening from leader to standby node
4. As per the follower health check, observing i/o timeout for connecting to the leader node on 443 port
06-05-2024 02:32 AM
Hi @Rupesh Sahu
can you check that with the next
'''
sv status etcd
sv stop etcd
sv start etcd
'''
and
'''
sv status etcd-proxy
sv stop etcd-proxy
sv start etcd-proxy
'''
06-05-2024 02:32 AM
Hi @Rupesh Sahu
are you trying to add the nodes to AF cluster and see this "initializing" status? Or does it happen once you've seeded the standbys and restored/configured them?
Have you used a backup file to deploy the leader or have you done that from scratch? Has this happened during an upgrade? What conjur version are you using?
Do you see any errors in the leader container log? for example, "hosts could not be resolved" or anything else?
We need more context and details on what you're doing to better understand this problem
06-05-2024 02:44 AM
etcd service was running. However, I have restarted the etcd service.
But etcd-proxy is not present.
sv status etcd-proxy
fail: etcd-proxy: unable to change to service directory: file does not exist
06-05-2024 03:07 AM
Hi @Rupesh Sahu
is it a new deployment? can you share more context about your env asked below in this thread?
06-05-2024 03:10 AM
Its not a new deployment. The deployment was not having any issue and everything was running fine.
When we rebooted the VM one by one for patching (yum update), after that this all issue arised.
06-05-2024 03:25 AM
@Rupesh Sahu
I suppose you didn't disable cluster service on nodes before VM patched/restarted, did you?
more details about this process are mentioned here
https://community.cyberark.com/s/article/Apply-OS-Updates-to-DAP-Auto-Failover-Nodes
Do you have a backup created before OS patching? If so, you can try to restore it and check the result
If no, we can't be 100% sure that the leader is still working correctly (in regards to AF cluster work), however, you can reset the Auto-failover Cluster manually by the next steps (just in case, do a backup of the current leader before doing the steps below):
---Stop the auto-failover service---
On each node in the auto-failover Leader cluster (Leader and Standbys), run the following command:
$ sv stop cluster
ok: down: cluster: 696s, normally up
--Reset the etcd system state--
On each node in the auto-failover Leader cluster (Leader and Standbys), run the following commands:
1) Stop `etcd` with the commands:
$ sv stop etcd
ok: down: etcd: 688s, normally up
$ sv stop etcd-proxy
ok: down: etcd-proxy: 1s, normally up
2) Reset the `etcd` data directory:
$ rm -rf /var/lib/etcd/*
3) Remove existing auto-failover configuration in the `/opt/conjur/etc/conjur.conf` file
Delete the lines that begin with `CLUSTER_` and `ETCD_` from the `/opt/conjur/etc/conjur.conf` file. These will be:
...
CLUSTER_NAME=...
CLUSTER_MACHINE_NAME=...
CLUSTER_MACHINE_ADDRESS=...
ETCD_INITIAL_CLUSTER=...
ETCD_INITIAL_CLUSTER_STATE=...
...
4) Remove existing auto-failover configuration in the `/etc/chef/solo.json` file.
Delete the lines that begin with `cluster_` from the `/etc/chef/solo.json` file. These will be:
...
"cluster_name": ...,
"cluster_master": ...,
"cluster_members": ...,
"cluster_machine_name": ...,
"cluster_machine_address": ...,
"cluster_ttl": ...,
"cluster_interval": ...
...
--Important--:
The `solo.json` file must be valid JSON after editing. Ensure removing these lines does not leave a dangling comma in the file.
In version 13.1 'solo.json' file can be found at location `/etc/cinc/solo.json`
--Re-establish auto-failover feature--
1) On the active Leader node, run the following command to enroll it in the new auto-failover cluster:
evoke cluster enroll \
--cluster-machine-name <leader-dns-name> \
<cluster-name>
For example:
evoke cluster enroll \
--cluster-machine-name conjur-master-1.mycompany.local \
conjur
2) On each Standby node, run the following command to enroll it in the new auto-failover cluster:
evoke cluster enroll \
--cluster-machine-name <standby-dns-name> \
--master-name <leader-dns-name> \
<cluster-name>
For example:
evoke cluster enroll \
--cluster-machine-name conjur-master-2.mycompany.local \
--master-name conjur-master-1.mycompany.local \
conjur
06-05-2024 03:26 AM
No I have not disabled the cluster service
06-05-2024 03:31 AM
@Rupesh Sahu
try to stop/rename the current standbys > do a backup of your leader > reset AF cluster on leader (see steps above) > reseed the standbys again and configure them again > check if all nodes are up/running/replicated > proceed with enrolling them to AF cluster
06-09-2024 10:09 PM
Thank you @Oleksandr Voloschuk It worked but I am facing some other issue. Will try to debug it. Thank you once again
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!

