Self-Inflicted AWS Auto Scaling Failures
Resolution
In AWS, the auto scaling process can fail for multiple reasons. Sometimes the cause for failure can actually be self inflicted. In this article, we will discuss some self-inflicted causes for auto-scaling failures in AWS. We will also discuss how to avoid these self-inflicted failures by following a few simple best practices.
Password changes
When a password is changed in your AWS VM-Series firewall, be sure it is not the account used by Lambda. Lambda makes API calls to the mgmt interface of the firewall. Upon API connectionn, Lambda sends commands which are critical for auto scaling completion. If you change the password used by Lambda, you will need to make changes in AWS to accommodate these changes. Below are two of the address objects that Lambda will need to update after every scaling event.
AWS-NAT-ILB
AWS-NAT-UNTRUST
If these objects are not updated, your traffic will not flow properly through the firewall, thus causing an outage. To correct this issue, you will need to change the API key used by the CloudFormation template.
1. First you will have to follow the steps on the link below to create the API key for the new password:
Get Your API Key
https://www.paloaltonetworks.com/documentation/71/pan-os/xml-api/get-started-with-the-pan-os-xml-api/get-your-api-key.html
2. Once your have generated the new API key, you will need to "Update your stack" to set the new API key. If you are not familiar with this process please follow the following article
How to Update Your AWS CloudFormation Deployment Without Relaunching Your Cloud
3. During the update process, you will add the new API key into the following location:
4. The account used in AWS in the "API Key for Firewall" section MUST have superuser access.
Old password in bootstrap.xml file
After changing your password, you must replace your bootstrap.xml file as well. If you don't change the password, when the firewall scales it will no longer be accessible due to incorrect password. In order to properly create a new bootstrap.xml file, we advise you to follow best practices. These best practices can also be found in the VM-Series Deployment Guide.
1. First you will need to save your configuration snapshot
2. Next you will need to export the named configuration snapshot
3. From there you will have to follow some key steps in order to avoid potential complications in AWS
Use the GitHub bootstrap files as seed
These changes are made to avoid potential conflict with AWS when it reissues the management interface IP address and SSL key
4. Once you make these changes rename the configuration snapshot to bootstrap.xml
5. Remove the previous bootstrap.xml file from the config directory in your S3 bucket
6. Upload the new bootstrap.xml file to the config directory in your S3 Bucket
By following these simple best practices, you will be sure to avoid the most common causes of auto scaling failures when using the Palo Alto Networks VM Series Firewall with AWS Auto Scale Groups.