Upgrading Your Linux Secrets Manager Credential Providers with Ansible at Enterprise Scale

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements

Welcome Idira to LIVEcommunity!

Our community just got stronger! Explore new identity security resources, connect with experts, and discover more ways to learn and collaborate.

Meet Idira

Idira Success Blogs
13 min read
L4 Transporter
Let's be honest, keeping End-Of-Life (EOL) versions in an environment is convenient. The production is stable (passwords are retrieved from the Vault for critical applications) and an upgrade of thousands of Providers can seem like a risky proposition. But remaining on versions that have ended their life is a bad idea and only delays the necessary step to take, upgrading them.

In this article, we’ll be covering the challenges with leaving EOL’d Providers, the challenges surrounding mass upgrades, CyberArk’s methodology for testing a mass upgrade process, and our recommended approach to upgrading them in bulk using Ansible.

The Risks and Implications
Continuing to use Credential Provider versions that have become EOL brings a multitude of unnecessary risks into an organization, ranging from operational challenges to security implications.

In this section we highlight five of the most critical risks and implications of using EOL components—risks that apply to any Idira component, not just Credential Providers.
  1. EOL versions, as the term suggests, are no longer supported or developed by CyberArk, which leads to supportability concerns.
  2. Similarly, if a security vulnerability or performance issue is discovered in an EOL version, it will not receive a fix, and Idira will encourage an upgrade to remediate the issue.
  3. EOL versions will not support the latest features, which is problematic as Credential Providers are components typically used by business units on target servers that may require those recent features.
  4. EOL versions are not supported on newer operating system versions as no testing has been performed, so unforeseen issues may arise.
  5. Last, but certainly not least, EOL versions are not tested with newer Vault versions, which can lead to unsupported configurations. Based on this, Idira documentation includes configuration considerations for Secrets Manager applications with the Vault. This comes in addition to the Recommendations for best performance document.
The Testing Environment
To simulate an environment with EOL’d Credential Providers, an architecture in AWS was built with a Primary Vault and its components running version 12.6. Dozens of Credential Providers were created using EC2 instance images provided by Amazon with different Linux distributions: Amazon Linux 2, RHEL8, and Ubuntu. The Credential Providers versions used for this test were 7.2 and 9.8 (source) and 12.6 (target).

In addition to this, an Ansible machine was built, and a Central Credential Provider deployed to secure sensitive passwords. The Central Credential Provider followed security best practices from this document.


Two accounts were created and onboarded to the Vault. These are the sensitive passwords referred to above.
  • provider_automation
    This internal EPV user was used to connect to the Vault during the CreateVaultEnvironment step and to perform the necessary actions, like creating additional system Safes or adding owners on Safes (e.g: the Provider user on the AppProviderConf Safe).
  • ansible
    This Linux account was used to connect to the Linux servers hosting the Credential Providers. It used an SSH key allowed in the authorized_keys file locally on each target. This user had the right permissions to use sudo. Storing this account in the Vault ensured that no one knew its password.
ka3Ht0000000q6lIAA__eid=ka3Ht0000000q6l&feoid=00N2J00000APFDu&refid=0EM2J000004RAnv.png
Figure 1: Architecture Diagram
Brainstorming
During this stage, several questions were raised by the Idira EMEA Security Services Team members. The goal was to make sure that we avoided common pitfalls or delays during our upgrade plan. This was mandatory in preparation for the Ansible Playbook development and implementation steps.
  • Which user is going to connect to the targets Linux servers?
  • Is it going to be a local or domain user?
  • Is it going to use a SSH Key or a password?
  • Once the user is logged in, how is it going to switch to root; how do we retrieve the root password for each target?
  • How many Credential Providers can we upgrade in one batch? 
  • If the OS on the target server (where the source Credential Provider (7.2 or 9.8) is deployed) is old and not supported by the target Credential Provider version (12.6), what do we do?
  • Should we split targets based on the Linux distribution?
  • Should we mix targets with different distributions (RHEL, Amazon Linux, Ubuntu...)?
  • Do all Linux distributions have a utility per default to unzip the Credential Provider installation package?
  • Do we really need to create the Vault environment when upgrading Credential Providers?

The Challenges
Multiple challenges can be faced when thinking of the practical steps required to upgrade Credential Providers:
  • For the automation tools being used, are the Subject Matter Experts available to help the PAM team? Or should we train some members of the PAM team to be autonomous with automation tools?
  • What kind of rights can we have to connect to the Linux target servers?
  • What kind of user can be used to connect to the Linux target servers?
  • No environment is the same, therefore it is impossible to build a one-fit-for-all solution.
  • If a third-party application was using a Credential Provider SDK that is older than 12.1, it needs to be upgraded to 12.1 or later, otherwise it will stop working.
  • How do we find all the Credential Providers deployed, and each Provider’s version and OS version?

The Flowchart
Following the brainstorming session, a flowchart was designed to demonstrate what the Ansible playbook would do against the Idira PAM service and the Linux systems:
ka3Ht0000000q6lIAA__eid=ka3Ht0000000q6l&feoid=00N2J00000APFDu&refid=0EM2J000004RAnw.png
Figure 2: Ansible Playbook Flowchart

Our Step-by-Step Testing Process
To answer the questions raised during the brainstorming process and outlined in the challenges section, the Idira EMEA Security Services Team went through the following step-by-step process, proving out the approach that would be needed.


Clearly, each Credential Providers upgrade project is going to be different depending on the environment. We hope our experience will help you figure out the particularities of your environment and deployment process.

Step 1: Credential Providers Discovery
Before even writing the Ansible playbook, we needed to know which Credential Providers versions were out there and on what kind of machines they were installed (Windows, Linux, other), and even more specifically for Linux, we needed know their Linux distribution and version.

This was done for two reasons:
  • To check the compatibility of the platform with the 12.6 version of the Credential Provider.
  • To provide us with data to design our upgrade plan and group the Providers by type (by specific business application or by specific Linux distribution and version).
To find the Credential Providers versions and related IPs, we used the Idira System Health script, found on the Idira Marketplace or Idira GitHub. This script provides an output of a .csv file with the list of Credential Providers machines and versions. Using this script, we could also see if any Providers had been decommissioned, but the Provider user was not deleted from the Vault. This can happen when someone deletes the target server hosting the Credential Provider (a cleaning would need to be performed in this case). With the list of IPs, we could use Ansible to get information about the OS and version. This was done using what is called ansible_facts. The output could also be saved as .csv file. Organizations who have a well-maintained CMDB can leverage that data to assist with the discovery effort.

By the end of this step, we were able to know what needed to be upgraded (source and target versions) and had a good view of the Credential Providers that were deployed and were active.

Step 2: Prechecks
To write the Ansible playbook that would execute the upgrade process, we first reviewed the official documentation for upgrading a Linux Credential Provider.

In our case, we decided to have a local user named “ansible” (with SSH Key authentication) on each Linux target. Its key was allowed in the authorized_keys file. This is how we would enable Ansible to remotely authenticate to each Linux target.


But before copying the installation files to the targets, we needed to gather several pieces of information:
  • The OS family. Is it Ubuntu or RHEL/CentOS/SLES/Oracle Linux?
  • Does /var/opt/CARKaim/.version_info exists on the machine already?
  • Has the aimprv service stopped correctly?
  ka3Ht0000000q6lIAA__eid=ka3Ht0000000q6l&feoid=00N2J00000APFDu&refid=0EM2J000004RAnx.png
Figure 3: Ansible Playbook Upgrade Playbook

Here are some outputs of the Ansible playbook:
ka3Ht0000000q6lIAA__eid=ka3Ht0000000q6l&feoid=00N2J00000APFDu&refid=0EM2J000004RAny.png
Figure 4: Ansible Playbook Output Example

Step 3: Staging the CP Package
In this step, we wanted to copy the 12.6 CP package to the target depending on the Linux distribution we defined in our prereq checks (Ubuntu, Amazon Linux, or RHEL target). This helped our playbook know which package to deploy to the target.
ka3Ht0000000q6lIAA__eid=ka3Ht0000000q6l&feoid=00N2J00000APFDu&refid=0EM2J000004RAnz.png
Figure 5: Ansible Playbook File Transfer Logic
 
Ansible has a built-in module for copying the files to the targets. The “ansible” user password that we used to run sudo commands on the Linux targets was retrieved from the Vault using a Central Credential Provider and the official Ansible Galaxy PAS module.

Note that for security reasons, the sudo permissions for this user should be limited to the required commands only.

Step 4: Unzip the Package
Next, we unzipped the 12.6 CP installation package locally on the target system.

We noticed that our RHEL / Amazon Linux targets had the capability to do this out of the box. But for Ubuntu targets, the unzip command was not found, so we had to install the module to execute the command.

Step 5: Configure the Vault.ini and aimparms Files
On the target server, we configured the Vault.ini file in the installation directory and the aimparms file under /var/tmp.

In our Ansible playbook we had a section named “Update parameters in /var/tmp/aimparms” to set the values in the aimparms:      
        replace:
          dest: /var/tmp/aimparms
          regexp: "{{item.regexp}}"
          replace: "{{item.replace}}"
        with_items:
          - {regexp: '^AcceptCyberArkEULA=.*$', replace: 'AcceptCyberArkEULA=Yes'}
          - {regexp: '^CredFilePath=.*$', replace: 'CredFilePath=/tmp/AAM_upgrade/user.cred'}
          - {regexp: '^LicensedProducts=.*$', replace: 'LicensedProducts=AIM'}
          - {regexp: '^.?CreateVaultEnvironment=.*$', replace: 'CreateVaultEnvironment=Yes'}
          - {regexp: '^VaultFilePath=.*$', replace: 'VaultFilePath=/tmp/AAM_upgrade/Vault.ini'}

As described above, we accepted the Licensing Agreement from CyberArk, specified the cred file path for the “provider_automation” user, specified the LicensedProducts, specified that we would create the Vault environment using the “provider_automation” user, and provided the path to the Vault.ini file.
Note that LicensedProducts can hold different values depending on the Credential Provider version: CP, AIM, OPM, ALL. Be sure to test the upgrade with different Credential Providers target versions if multiples are in scope. For instance, a Credential Provider can use version 12.0 maximum due to an old SDK used by the third-party Marketplace integration at the moment, otherwise it would break based on Provider SDK applications failing after upgrade to 12.1.

At this stage, we discussed the CreateVaultEnvironment step. Could we set it to No in order to simplify the overall upgrade?


Well, take a look at the results when CreateVaultEnvironment=No in the aimparms file. We did a manual test without the Ansible Playbook and the Provider was upgraded successfully from 9.8 to 12.6.
ka3Ht0000000q6lIAA__eid=ka3Ht0000000q6l&feoid=00N2J00000APFDu&refid=0EM2J000004RAo0.png
Figure 6: Confirmation of successful upgrade (System Health PVWA)

The problem was that the Provider was still using the old 9.8 conf file under AppProviderConf. The 12.6 was not generated in the Safe and did not exist locally.

ka3Ht0000000q6lIAA__eid=ka3Ht0000000q6l&feoid=00N2J00000APFDu&refid=0EM2J000004RAo1.png
Figure 7: AppProviderConf Safe and Provider config files

Also, the 9.8 configuration file was missing some parameters added in newer CP versions such as AutomaticProviderPasswordRefreshInterval and KeyStorage.


ka3Ht0000000q6lIAA__eid=ka3Ht0000000q6l&feoid=00N2J00000APFDu&refid=0EM2J000004RAo2.png
Figure 8: V9.8 File vs. V12.6 File with Differences Highlighted

And lastly, the AppProviderCacheSafe was not created when it should have been.

ka3Ht0000000q6lIAA__eid=ka3Ht0000000q6l&feoid=00N2J00000APFDu&refid=0EM2J000004RAo3.png
Figure 9: Top Screenshot Showing Missing AppProviderCacheSafe

Step 6: Create Provider Automation User Cred File User
Based on this testing, we concluded that the CreateVaultEnvironment must be set to Yes in the aimparms file as stated in the official upgrade documentation. To perform the CreateVaultEnvironment process, we created a specific user named "provider_automation" based on the necessary permissions listed in the official documentation (pre-installation steps section). We reused some sections of the “Provider Installation” Ansible Playbook.

From there, we changed the permissions on the CreateCredFile utility and generated a cred file for the provider_automation user we just created. The “provider_automation” was an EPV user in the Vault. This account could be rotated by the CPM using the Idira Vault platform.

Step 7: Upgrade the Credential Provider
Depending on the OS family, we used either the yum (Amazon Linux / RHEL) or apt (Ubuntu) package managers included with Ansible core.


To develop the Ansible playbook, we reviewed the official documentation and played manual upgrades with and without the CreateVaultEnvironment option.

Step 8: Restart the aimprv Service
Once the Credential Provider was upgraded, we restarted the aimprv service as simply starting it did not always work on Ubuntu.


We also recommend reviewing the APPConsole.log under /var/opt/CARKaim/logs. The following line should be displayed:
APPAP035I Application Password Provider [Prov_name] on machine [IP] version [12.6] is up [AIM mode] and working with Vault [Vault_name]


Upgrade Scale Considerations
One of the primary requirements of an organization-wide upgrade is to control and verify that the upgrade is successful. Upgrading in batches helps control the upgrade flow and rollout. If any upgrades are not successful, it’s much easier to roll back in an efficient manner and not impact production for too much time. Taking this into consideration, it would be unrealistic to execute the Ansible Playbook on more than 50 targets at the same time.


Depending on the size of your environment, consider incorporating line of business or other organizational factors into your batching or prioritization to make it easier to roll out (or roll back).

Next Steps to Improving Your Knowledge About Credential Providers
To summarize, these are the key stages that should be taken into account when developing your own upgrade process:
  1. Brainstorm about what is allowed or not allowed in your company, the automation tools used, how you can connect to the Credential Providers machines. Involve the right people from the start.
  2. Know your environment well: map your Credential Provider versions, OSs, and business applications.
  3. Define which Credential Provider version should be the target.
  4. Build the automation logic around Idira documented steps for upgrading a Credential Provider.
Using the logic and process above, you should feel confident in developing your own enterprise-wide Credential Provider upgrade plan for Linux systems for your organization.

Credential Providers have many secrets (no pun intended!), and if you want to go further with them, we suggest reading the following knowledge articles. These articles cover a variety of topics about Credential Providers, including internal user password rotation, timeouts, C3 integrations, SDK compability issues, service not starting, log locations, and enabling debugging.

If you need additional guidance on developing your enterprise-wide upgrade process, feel free to talk to your Account Team about requesting Idira Security Services involvement.

Furthermore, if you’re looking for guidance on developing an enterprise-wide secrets management strategy, securing your 3rd party security tools or homegrown OS-based applications, we encourage you to explore other Success Blog articles.
  • 4316 Views
  • 0 comments
  • 0 Likes
Top Liked Authors