AWS PAN-OS 11 Interfaces never become active

[DGentry]

I'm trying to bring up a new PAN-OS 11.1 instances in AWS, installed from aws-marketplace/PA-VM-AWS-11.1.0-f1260463-68e1-4bfb-bf2e-075c2664c1d7. I am able to reach the management IP address, both SSH and the web UI are working. However the two intended network interfaces never appear in "show interface all" nor in the UI Network > Interfaces > Ethernet.

I created three subnets within the VPC and three Elastic Network Interfaces, which are attached to the EC2 instance.

• The eni used for the management interface and for the WAN have Elastic IP addresses attached.
• The subnets for MGMT and LAN have a routing table with a default route pointing to the ENI.
• The subnet for the WAN has a routing table with a default route pointing to the Internet Gateway for the VPC.

From the AWS EC2 instance tab:

Interface ID	Description	IPv4 Prefixes	IPv6 Prefixes	Public IPv4 address	Private IPv4 address	Attachment status	VPC ID	Subnet ID	Source / destination check	Security groups	Interface type
eni-09c...	MGMT	–	–	52.25.x.y	10.0.6.71	attached	vpc-0d2...b90	subnet-036...	enabled	sg-093...	Elastic network interface
eni-062...	WAN	–	–	35.82.x.y	10.0.64.130	attached	vpc-0d2...b90	subnet-025...	disabled	sg-083...	Elastic network interface
eni-06b...	LAN	–	–	–	10.0.137.103	attached	vpc-0d2...b90	subnet-03c...	disabled	sg-07f...	Elastic network interface

--------

In "show system state" I see the MAC addresses of the Elastic Network Interfaces I expect. sys.s1.p1.hwaddr is the MAC address of eni-062... intended for the WAN, and sys.s1.p2.hwaddr is the MAC address of eni-06b... intended for the LAN.

admin@PA-VM> show system state
…
sys.s1.p1.bus: 0000:00:06.0
sys.s1.p1.capability: [ auto, 10Mb/s-half, 10Mb/s-full, 100Mb/s-half, 100Mb/s-full, 1Gb/s-half, 1Gb/s-full, 10Gb/s-half, 10Gb/s-full, 25Gb/s-half, 25Gb/s-full, 40Gb/s-half, 40Gb/s-full, 100Gb/s-half, 100Gb/s-full, ]
sys.s1.p1.cfg: { 'breakout': False, 'fec': 0, 'mode': Disabled, 'pause-frames': True, 'setting': auto, }
sys.s1.p1.detail: { }
sys.s1.p1.driver: net_ena
sys.s1.p1.eni:
sys.s1.p1.hwaddr: 06:71:1a:54:54:9d
sys.s1.p1.mtu: 1504
sys.s1.p1.phy: { 'link-partner': { }, 'media': CAT5, 'type': Ethernet, }
sys.s1.p1.rate: { 'duration': 28560, 'last-sample': 2023-12-23 22:18:40, 'rx-broadcast': 0, 'rx-bytes': 0, 'rx-multicast': 0, 'rx-unicast': 0, 'tx-broadcast': 0, 'tx-bytes': 0, 'tx-multicast': 0, 'tx-unicast': 0, }
sys.s1.p1.state: board_port_autoneg
sys.s1.p1.stats: { 'link-down': 0, 'rx-broadcast': 0, 'rx-bytes': 22824, 'rx-discards': 0, 'rx-error': 0, 'rx-missed-error': 0, 'rx-multicast': 0, 'rx-unicast': 523, 'tx-broadcast': 0, 'tx-bytes': 0, 'tx-error': 0, 'tx-multicast': 0, 'tx-unicast': 0, }
sys.s1.p1.status: { 'link': Down, 'mode': Disabled, 'pause-frames': True, 'setting': Unknown, 'type': RJ45, }
…
sys.s1.p2.bus: 0000:00:07.0
sys.s1.p2.capability: [ auto, 10Mb/s-half, 10Mb/s-full, 100Mb/s-half, 100Mb/s-full, 1Gb/s-half, 1Gb/s-full, 10Gb/s-half, 10Gb/s-full, 25Gb/s-half, 25Gb/s-full, 40Gb/s-half, 40Gb/s-full, 100Gb/s-half, 100Gb/s-full, ]
sys.s1.p2.cfg: { 'breakout': False, 'fec': 0, 'mode': Disabled, 'pause-frames': True, 'setting': auto, }
sys.s1.p2.detail: { }
sys.s1.p2.driver: net_ena
sys.s1.p2.eni:
sys.s1.p2.hwaddr: 06:62:fb:e5:5e:9f
sys.s1.p2.mtu: 1504
sys.s1.p2.phy: { 'link-partner': { }, 'media': CAT5, 'type': Ethernet, }
sys.s1.p2.rate: { 'duration': 28560, 'last-sample': 2023-12-23 22:18:40, 'rx-broadcast': 0, 'rx-bytes': 0, 'rx-multicast': 0, 'rx-unicast': 0, 'tx-broadcast': 0, 'tx-bytes': 0, 'tx-multicast': 0, 'tx-unicast': 0, }
sys.s1.p2.state: board_port_autoneg
sys.s1.p2.stats: { 'link-down': 0, 'rx-broadcast': 0, 'rx-bytes': 21252, 'rx-discards': 0, 'rx-error': 0, 'rx-missed-error': 0, 'rx-multicast': 0, 'rx-unicast': 506, 'tx-broadcast': 0, 'tx-bytes': 0, 'tx-error': 0, 'tx-multicast': 0, 'tx-unicast': 0, }
sys.s1.p2.status: { 'link': Down, 'mode': Disabled, 'pause-frames': True, 'setting': Unknown, 'type': RJ45, }

However no interfaces appear in "show interface all" and the Web UI never shows their status as green.

admin@PA-VM> show interface all

total configured hardware interfaces: 0

name id speed/duplex/state mac address
--------------------------------------------------------------------------------

aggregation groups: 0


total configured logical interfaces: 0

name id vsys zone forwarding tag address
------------------- ----- ---- ---------------- ------------------------ ------ ------------------

--------

In other posts I've read that this means the interface is not configured. I set the interface type of the first two Ethernet interfaces to Layer3, created a management profile which allows ICMP ping, and set their IP address to use DHCP.

The ENI which I'm intending as the WAN interface has a public IPv4 Elastic IP address associated with it, which I would expect means AWS should respond to a DHCP request for that interface at least.

--------

I've rebooted the EC2 instance multiple times, including going all the way to Stopping the instance and then Starting it again to ensure any new device tree will be properly handled at boot.

I'm running out of ideas of what to try. What else could be preventing PAN from seeing these links as configured and active?

[DGentry]

This was marked as spam when posted, I've been trying other things for a couple days.

One thing was to shut down the PAN-OS 11.1 VM and start up an EC2 instance with one of the earlier bundles running PAN-OS 9. This one is able to boot the image but I cannot log in, either via SSH nor web.

• SSH appears to be because we require Instance Metadata v2, which breaks the SSH authorized_key support in the PAN-OS 9 instance. https://live.paloaltonetworks.com/t5/vm-series-in-the-public-cloud/new-vm-asks-for-password-using-ss...
• The instructions say to use SSH to change the admin password before trying to log in using the WebUI, which I couldn't do because of this. Web UI login rejects admin/admin. They might change the default password in the AMI images, to protect people from leaving an open router login on an AWS address.

PAN-OS 11.1 is apparently able to use instance metadata v2 to handle its SSH authorized_keys. However it does make me wonder if the interfaces not coming up is somehow related to instance metadata. I don't currently have an environment with metadata v1 supported to try it.

An EC2 instance can retrieve information about elastic network interfaces using: http://169.254.169.254/latest/meta-data/network/interfaces/
I'll be that is what the PAN VM Series does. With metadata v2, that code would need to know how to fetch and add a X-aws-ec2-metadata-token
They appear to have added this support for SSH authorized_keys, but I bet the Interface handling code does not.

[DGentry]

Confirmed that the lack of metadata v1 is part of the problem. I've enabled instance metadata v1 on the PAN-OS 11.1 instance, and now it is a step closer: debug show vm-series interfaces all now shows the correct Elastic Network Interface. Previously, the Eni column was blank. The interfaces still don't come up, 

admin@PA-VM> debug show vm-series interfaces all
Interface Base-OS_port    Base-OS_MAC        PCI-ID   Driver  Eni
 mgt          eth0  06:61:22:d2:24:f9  0000:00:05.0      ena
 Ethernet1/1  eth1  06:10:fd:9c:14:23  0000:00:06.0  net_ena  eni-0f6500e5
 Ethernet1/2  eth2  06:7e:43:5b:b9:35  0000:00:07.0  net_ena  eni-05a70efe
admin@PA-VM>

[DGentry · Accepted solution]

Confirmed that the lack of metadata v1 is the problem. After enabling instance metadata v1 on the PAN-OS 11.1 VM,debug show vm-series interfaces all shows the correct Elastic Network Interface. Previously, the Eni column was blank.

After committing the config again and rebooting again, success! The links came up.

admin@PA-VM> show interface all

total configured hardware interfaces: 2

name         id  speed/duplex/state   mac address
-------------------------------------------------------
ethernet1/1  16  ukn/ukn/up           06:10:fd:9c:14:23
ethernet1/2  17  ukn/ukn/up           06:7e:43:5b:b9:35

aggregation groups: 0


total configured logical interfaces: 2

name         id  vsys zone  forwarding   tag address
------------ --  ---- ----- ------------ --- ---------------
ethernet1/1  16  1    wan   vr:default   0   10.0.64.180/24
ethernet1/2  17  1    lan   vr:default   0   10.0.128.219/24