11-11-2015 01:17 PM
Hey guys,
Looking for some assistance on getting a strange issue resolved. I've got a site-to-site VPN set up for a connection to AWS for one of our customers. I've created two loopbacks, loopback.5 and loopback.6, on the outside zone that fall in the same subnet as our regular ethernet interface, which is a /29. I've verified that our peers IP's can reach each other successfully, so routing is good there. I've also had two techs from AWS look at the Palo config, everything is matching their side, they saw our traffic, our cookie ID, all that good stuff. I've entered in the PSK at least 6 times, and nothing is coming up. I also asked about the proxy ID's, just to make sure they weren't needed, tech said they don't use them. I decided to do a pcap just for grins to see what was actually happening, and from what I'm seeing, the Palo is taking the traffic from the AWS side and shunting it from loopback.5 to loopback.1 for whatever reason, where it gets dropped, because that's just wrong. Is there any explanation that you guys could think that would explain why the packets are being processed this way? I'm going to include some logs for you guys to review, I'm hesitant to upload the pcaps, but I can if it's helpful. Thanks a lot for looking, hope the answer jumps out at somebody!
Palo Alto 5050-PanOS 5.0.12
GwID Name Peer Address/ID Local Address/ID Protocol Proposals
-------- ---- --------------- ---------------- -------- ---------
1 ike-vpn-e0c72a89-0 xx.xx.xx.xx(ipaddr:xx.xx.xx.xx) xx.xx.xx.xx(ipaddr:xx.xx.xx.xx) Main [PSK][DH2][AES128][SHA1] 28800-sec
tunnel IPSec-Tunnel1
id: 4
type: IPSec
gateway id: 1
local ip: xx.xx.xx.xx
peer ip: xx.xx.xx.xx
inner interface: tunnel.3
outer interface: loopback.5
state: init
session: 12359
tunnel mtu: 1427
lifetime remain: N/A
monitor: off
monitor packets seen: 0
monitor packets reply: 0
en/decap context: 19494
local spi: 00000000
remote spi: 00000000
key type: auto key
protocol: ESP
auth algorithm: NOT ESTABLISHED
enc algorithm: NOT ESTABLISHED
proxy-id local ip: 0.0.0.0/0
proxy-id remote ip: 0.0.0.0/0
proxy-id protocol: 0
proxy-id local port: 0
proxy-id remote port: 0
anti replay check: yes
copy tos: no
authentication errors: 0
decryption errors: 0
inner packet warnings: 0
replay packets: 0
packets received
when lifetime expired:0
when lifesize expired:0
sending sequence: 0
receive sequence: 0
encap packets: 0
decap packets: 0
encap bytes: 0
decap bytes: 0
key acquire requests: 1429
phase-1 SAs
GwID/client IP Peer-Address Gateway Name Role Mode Algorithm Established Expiration V ST Xt Phase2
--------------- ------------ ------------ ---- ---- --------- ----------- ---------- - -- -- ------
1 xx.xx.xx.xx ike-vpn-e0c72a89-0 Init Main PSK/ NO/ TBD/ TBD v1 3 2 0
Show IKEv1 IKE SA: Total 1 gateways found. 1 ike sa found.
phase-2 SAs
GwID/client IP Peer-Address Gateway Name Role Algorithm SPI(in) SPI(out) MsgID ST Xt
--------------- ------------ ------------ ---- --------- ------- -------- ----- -- --
1 xx.xx.xx.xx:0 ike-vpn-e0c72a89-0 Init / / / / 00000000 00000000 00000000 0 0
Show IKEv1 phase2 SA: Total 1 gateways found. 1 ike sa found.
System log, I tried GoogleFu, not much luck other than passphrase:
2015/11/11 13:44:31 info vpn ike-vp ike-neg 0 IKE phase-1 negotiation is failed as initiator, main mode. Failed SA: xx.xx.xx.xx[500]-xx.xx.xx.xx[500] cookie:bf1b32ec25b3acd5:0000000000000000. Due to timeout.
2015/11/11 13:44:31 info vpn ike-vp ike-neg 0 IKE phase-1 SA is deleted SA: xx.xx.xx.xx[500]-xx.xx.xx.xx[500] cookie:bf1b32ec25b3acd5:0000000000000000.
2015/11/11 13:44:33 info vpn ike-vp ike-neg 0 IKE phase-1 negotiation is started as initiator, main mode. Initiated SA: xx.xx.xx.xx[500]-xx.xx.xx.xx[500] cookie:d5290c9ce7032f8d:0000000000000000.
11-12-2015 08:09 AM
Hi John,
Can you please confirm if the AWS has a static IP address configured or if you obtain an IP address dynamically. The reason why I ask this is because based on type of addressing AWS tends to pick up either v1 or v2 of IKE. Additionally can you clarify where the drop was seen and if there was a rule listed in the deny log. The reason why I ask this is because when traffic is generated from an outside loopback going to another interface on the outside zone, I would expect to either have the implicit "allow intrazone" rule enabled or have an explicit rule to allow the traffic through. Please let me know if any of the two is present on the device.
HTH.
Warm Regards,
Kamalesh R
11-12-2015 04:46 PM
If you are using a loopback address that is in the same range as the Ethernet interface, are you sure that the provider router is finding the loopback via ARP? Since the address is local to the provider's address on that link, it is going to try to resolve it directly rather than route it to your firewall on the next-hop address.
FWIW, we terminate a bunch of VPNs on a loopback, and it works fine. The loopback is "behind" the firewall from the provider router's point of view, so it routes to the firewall as the next hop.
11-13-2015 08:34 AM
Hey Kamalesh,
Thanks for the reply. I'm uncertain of the config on the AWS side of things, that's another company configuring that, but I'll pose that question to them. I've looked in the logs to see if there were any deny's from the addresses for the AWS and there were not. I don't in fact see any mention of the traffic going to the other loopback in the logs, only in the pcaps I took. There should be an intrazone rule allowing traffic however, so I don't believe that's the case. I'll do some more digging to see.
11-13-2015 08:39 AM
We've confirmed end to end communication between the peers through ping and traceroute, but ARP wouldn't come into play until the tunnel is up, which isn't happening past phase 1. There's no NAT involved here either, I'm on a public IP on my side. I've followed the example setup here for doing a site to site using loopbacks, as well as referenced Keith Barker's CBT Nuggets and another YouTube video to get things working. Everyone that's seen the config on the firewall has stated it appears to be correct, and that include the AWS tech that has done this very thing many times with the Palo's himself. I'm hoping to get our support contract updated and get someone from Palo to take a look, I'm stumped!
11-24-2015 02:21 AM
Hi John,
Can you please tell me if there is a specific reason for using a loopback in the first place? The reason why I ask this of you is two-fold:
+ We have many successful connections done from a physical ethernet interface and that would probably be preferable to using a loopback (to avoid dataplane-like processing of VPN negotiation packets)
+ When using a single encapsulation (IPSec in this case), a separate termination point is not really required for the traffic to work. If the IP address is a concern, then you can try add additional IP addresses to the same interface for this sake.
The major reason why I am suspecting internal traffic processing causing an issue is because of the fact as an Initiator, I would expect the init SPI value to be filled in if atleast the first packet is sent out. The fact that the init SPI is also 0 when looking at the tunnel, indicates that there may be a problem with even sending the first packet. Considering that you have already looked at the configuration with an AWS tech, chances are there is a problem on the underlying medium (internet path).
Additionally, if this was previously working fine, then there is a possibility that the ISAKMP FSM or keys may have degraded causing a state mismatch during rekey, thereby failing the negitations. If this should b the case, then issuing a "clear vpn ike-sa" and "clear vpn isakmp-sa" would help with the issue.
HTH.
Warm Regards,
Kamalesh R
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!
