HA on AWS Using a Secondary IP

[NathanielM]

Hi,

Just checking if anyone has successfully deployed the latest HA mode "secondary-ip". Unfotunately the deployment guides can be described more as "guides" rather than detailed instructions. Furthermore they are fragmented so one has to scramble over different places and review pages, sometimes unrelated to the new mode 😅.

Anyway my issue is HA is up and running, and I presume in a ready state due to the CLI output. In fact I am even able to suspend a device and the other assumes the active role. However the big issue is that the actual "vm_series" plugin for AWS magic does not happen:

1. No secondary IPs move over to the passive device.
2. Route tables do not change to the ENI of the passive device.

I have tried with plugin 2.1.5 and 2.1.7 and it is the same behaviour. PAN-OS 10.1.5-h2, VM-300, m5.xlarge, eu-west-1.

Anyone have any tips?

By the way, what wasn't specified in the guides is that the management interface needs Internet access in order to run some of the "show plugin" commands.

Topology

💡The test instance can reach the Internet through the active firewall which performs source NAT to the secondary IP.

Active:

PA-VM(active)> show plugins vm_series aws ha failover-mode

HA failover mode: secondary-ip

PA-VM(active)> show plugins vm_series aws ha state

Type                     Active                                                      Passive                                                Status
======================================================================================================================================================
INTERFACES               0: eni-0721c7f1fd0d6d5c3                                    0: eni-0118f5b32a378a738                               Pass
                         1: eni-09f1b2f0bb79ceb93                                    1: eni-0b4aa2715fe380995
                         2: eni-0440268dd0a05e96c                                    2: eni-0188d3f804ea62a23
                         3: eni-00b98967a33415e66                                    3: eni-0e41883adf46de89d
______________________________________________________________________________________________________________________________________________________
IAM_POLICY_PERMISSIONS   ec2:AttachNetworkInterface                                  ec2:AttachNetworkInterface                             Pass
                         ec2:DetachNetworkInterface                                  ec2:DetachNetworkInterface
                         ec2:DescribeInstances                                       ec2:DescribeInstances
                         ec2:DescribeNetworkInterfaces                               ec2:DescribeNetworkInterfaces
                         ec2:AssignPrivateIpAddresses                                ec2:AssignPrivateIpAddresses
                         ec2:AssociateAddress                                        ec2:AssociateAddress
                         ec2:DescribeRouteTables                                     ec2:DescribeRouteTables
                         ec2:ReplaceRoute                                            ec2:ReplaceRoute
______________________________________________________________________________________________________________________________________________________
INSTANCE_ID              i-022250b4dd95b5d60                                         i-079b89223e9372e15                                    -
______________________________________________________________________________________________________________________________________________________
HA_FAILOVER_MODE         secondary-ip                                                secondary-ip                                           Pass
______________________________________________________________________________________________________________________________________________________
IAM_POLICY_NAMES         single-az-vpn-labs-fw-ha_interface-swap_route-edit-profile  single-az-vpn-labs-fw-ha_interface-swap_route-edit-profile-

______________________________________________________________________________________________________________________________________________________
IAM_ROLE                 single-az-vpn-labs-fw-ha_interface-swap_route-edit-role     single-az-vpn-labs-fw-ha_interface-swap_route-edit-role-
______________________________________________________________________________________________________________________________________________________

PA-VM(active)> show plugins vm_series aws ha ips

Interface                Eni-Id                             PrimaryIP:PublicIP            SecondaryIP:PublicIP
========================================================================================================================
Management               eni-0721c7f1fd0d6d5c3              10.65.48.10:
------------------------------------------------------------------------------------------------------------------------
Ethernet1/1              eni-09f1b2f0bb79ceb93              10.65.48.40:
------------------------------------------------------------------------------------------------------------------------
Ethernet1/2              eni-0440268dd0a05e96c              10.65.48.75:                  10.65.48.74:3.248.70.122
------------------------------------------------------------------------------------------------------------------------
Ethernet1/3              eni-00b98967a33415e66              10.65.48.139:                 10.65.48.138:
------------------------------------------------------------------------------------------------------------------------

PA-VM(active)>

Passive:

PA-VM(passive)> show plugins vm_series aws ha failover-mode

HA failover mode: secondary-ip

PA-VM(passive)> show plugins vm_series aws ha state

Type                     Active                                                      Passive                                                Status
======================================================================================================================================================
INTERFACES               0: eni-0118f5b32a378a738                                    0: eni-0721c7f1fd0d6d5c3                               Pass
                         1: eni-0b4aa2715fe380995                                    1: eni-09f1b2f0bb79ceb93
                         2: eni-0188d3f804ea62a23                                    2: eni-0440268dd0a05e96c
                         3: eni-0e41883adf46de89d                                    3: eni-00b98967a33415e66
______________________________________________________________________________________________________________________________________________________
IAM_POLICY_PERMISSIONS   ec2:AttachNetworkInterface                                  ec2:AttachNetworkInterface                             Pass
                         ec2:DetachNetworkInterface                                  ec2:DetachNetworkInterface
                         ec2:DescribeInstances                                       ec2:DescribeInstances
                         ec2:DescribeNetworkInterfaces                               ec2:DescribeNetworkInterfaces
                         ec2:AssignPrivateIpAddresses                                ec2:AssignPrivateIpAddresses
                         ec2:AssociateAddress                                        ec2:AssociateAddress
                         ec2:DescribeRouteTables                                     ec2:DescribeRouteTables
                         ec2:ReplaceRoute                                            ec2:ReplaceRoute
______________________________________________________________________________________________________________________________________________________
INSTANCE_ID              i-079b89223e9372e15                                         i-022250b4dd95b5d60                                    -
______________________________________________________________________________________________________________________________________________________
HA_FAILOVER_MODE         secondary-ip                                                secondary-ip                                           Pass
______________________________________________________________________________________________________________________________________________________
IAM_POLICY_NAMES         single-az-vpn-labs-fw-ha_interface-swap_route-edit-profile  single-az-vpn-labs-fw-ha_interface-swap_route-edit-profile-

______________________________________________________________________________________________________________________________________________________
IAM_ROLE                 single-az-vpn-labs-fw-ha_interface-swap_route-edit-role     single-az-vpn-labs-fw-ha_interface-swap_route-edit-role-
______________________________________________________________________________________________________________________________________________________

PA-VM(passive)> show plugins vm_series aws ha ips

Interface                Eni-Id                             PrimaryIP:PublicIP            SecondaryIP:PublicIP
========================================================================================================================
Management               eni-0118f5b32a378a738              10.65.48.11:
------------------------------------------------------------------------------------------------------------------------
Ethernet1/1              eni-0b4aa2715fe380995              10.65.48.41:
------------------------------------------------------------------------------------------------------------------------
Ethernet1/2              eni-0188d3f804ea62a23              10.65.48.76:
------------------------------------------------------------------------------------------------------------------------
Ethernet1/3              eni-0e41883adf46de89d              10.65.48.140:
------------------------------------------------------------------------------------------------------------------------

PA-VM(passive)>

[NathanielM · Accepted solution]

TAC resolved this for me, it was the dpdk settings for anyone else facing the same issue. 

Interestingly the guide for "interface move" ha mode explains this needs to be turned off but not the "secondary IP" guide.

Before:

PA-VM(active)> show system setting dpdk-pkt-io

Device current Packet IO mode:                 DPDK
Device DPDK Packet IO capable:                 yes
Device default Packet IO mode:                 DPDK

After:

PA-VM(active)> show system setting dpdk-pkt-io

Device current Packet IO mode:                 Packet MMAP
Device DPDK Packet IO capable:                 yes
Device default Packet IO mode:                 Packet MMAP

Here is a good explanation of DPDK:

https://docs.paloaltonetworks.com/plugins/vm-series-and-panorama-plugins-release-notes/vm-series-plu...

I think it was off by default until recently hence the guide may be out of date.