topic Re: What did I miss? Cross-zone/vpn traffic in General Topics

What did I miss? Cross-zone/vpn traffic

Nathan.S — Fri, 12 Apr 2013 14:25:23 GMT

New PA200 installed and working on getting it setup. Aside from a 2wk demo, I have little experience with PAN.

I've got a Site-To-Site VPN configured to an ASA5505 at another of our offices.

I have one zone setup for a Wifi network. (Called Wifi) IP space behind that zone is 172.168.1.0/24. Interface 1/3 is configured with the IP 172.168.1.1 PAN is providing DHCP for this network.

Client behind this network has a DHCP address of 172.168.1.10 with a subnet mask of 255.255.255.0

On the other side of tunnel.1 (this interface is tagged as the Remote zone) is the IP space 10.5.0.0/16. Client behind that firewall has IP 10.5.1.25/255.255.0.0.

I have Security Policies configured to allow traffic between the zones.

However, pinging between the clients routes through the wrong rule (Default rule allowing outside access) instead of the Wifi-to-Remote or Remote-To-Wifi rules (depending on which side I'm pinging from)

I can ping from the CLI of the PAN to any client in the Remote zone.

And I can ping from any client in the Remote zone to the 172.168.1.1 address of the PAN.

Since it wasn't working with the zones defined, I changed the Source/Destination of the rules to the specific IP ranges of the zones in the security rules.

Wifi-To-Remote:

Source: 172.168.1.0/24

Destination: 10.5.0.0/16

Remote-To-Wifi:

Source: 10.5.0.0/16

Destination: 172.168.1.0/24

(I've also tried putting in the range directly, 172.168.1.1-172.168.1.254, with the same result)

Now in the Monitor->Traffic tab, the Wifi-Client to Remote-Client (and vice versa) ping shows up in the correct rule, but the ping still doesn't complete (Request Timed Out)

I'm sure there's something relatively simple that I've missed, could someone point it out for me?

Re: What did I miss? Cross-zone/vpn traffic

craymond — Fri, 12 Apr 2013 14:55:39 GMT

Have you added the static routes in the VR's routing table?

Re: What did I miss? Cross-zone/vpn traffic

Nathan.S — Fri, 12 Apr 2013 14:58:20 GMT

Two static routes in the VR:

0.0.0.0/0 with ISP gateway as next hop on outside-facing interface

10.5.0.0/16, no next hop, tunnel.1 interface

Edit to add: I tried adding a next hop of 10.5.1.1 (Remote gateway) to no avail.

Re: What did I miss? Cross-zone/vpn traffic

craymond — Fri, 12 Apr 2013 20:10:44 GMT

Assuming you have the IPSEC Tunnel, IKE Gateway, IPSEC Crypto, IKE Crypto configured correctly.... This looks to be a routing issue. What does a tracert look like from a client on one side to a client on the other side? Where does it stop?

Re: What did I miss? Cross-zone/vpn traffic

NGS_SOC — Fri, 12 Apr 2013 21:54:47 GMT

Does the Zones belog to the same virtual router? If not, even with correct security rules, traffic won't flow.

In static route you need only net and interface tunnel.1, no next hop

Also pay attention to secuty acls, use zones, not the enforced network until connection is ok.

Re: What did I miss? Cross-zone/vpn traffic

apasupulati — Fri, 12 Apr 2013 22:16:07 GMT

Hey Nathan,

Is the remote side configured to send the traffic (destined to 172.168.1.x) through its tunnel interface as well?

You said: Since it wasn't working with the zones defined, I changed the Source/Destination of the rules to the specific IP ranges of the zones in the security rules.

Assuming you did have a static route for remote subnet pointing to the tunnel.1 interface (remote zone), the traffic must match the Wifi-to-Remote rule, unless the default outbound rule is above this specific rule ignoring it.

You can also try taking pcaps on the remote side and verify if they are receiving our packets and responding back to us.

Thanks,

Aditi

Re: What did I miss? Cross-zone/vpn traffic

Nathan.S — Mon, 15 Apr 2013 14:15:03 GMT

traceroute from the Remote site fails. Both too the PAN (172.168.1.1) and to the client (172.168.1.10). Even though ping to the PAN works.

from the local site to the remote site fails. No hops.

Since there's 'nothing' between the clients on each side of the VPN, I'm not surprised there isn't anything showing in a traceroute.

I am surprised that traceroute is failing from Remote to the PAN, since the ping works.

All the interfaces of the PAN are in a single VR. It's the only VR on the PAN.

The default rule to allow out-bound traffic is at the bottom of the stack. The rules to allow from local-to-remote and remote-to-local are located above that default rule.

Presently only have 3 rules in the PAN.

Cisco is routing traffic to 172.168.1.0/16 across the tunnel.

Re: What did I miss? Cross-zone/vpn traffic

craymond — Mon, 15 Apr 2013 19:10:49 GMT

In order to troubleshoot this, I would recommend creating a deny all rule at the bottom with alerts (any zone to any zone with just alerts) so you can see what is being implicitly denied. It sounds like you have 2 rules for traffic from "local" to "remote" and from "remote" to "local", and a third from "local" to "external". With those rules, you will not be able to view the implicit denies in the traffic log, so it will be harder to troubleshoot unless you want to do it from the CLI.

Re: What did I miss? Cross-zone/vpn traffic

Nathan.S — Mon, 15 Apr 2013 19:17:36 GMT

I'll add that rule.
I don't mind working via the CLI, just don't know my way around there.

Re: What did I miss? Cross-zone/vpn traffic

Nathan.S — Mon, 15 Apr 2013 22:24:12 GMT

Adding the Deny-All rule seems to have broken everything, but I'm not able to ping from either side across the VPN either(even after disabling the Deny-All rule, just to test). Not sure what I broke there.

Re: What did I miss? Cross-zone/vpn traffic

craymond — Tue, 16 Apr 2013 14:05:19 GMT

The Deny rule at the end of the allows should not have broken anything if you have the correct allow rules above it. As I'm sure you know, the rules are applied in order, and there is an implicit deny rule at the end that blocks everything that is not explicitly allowed. If your rules are:

src-zone: WiFi --> dst-zone: Remote, any source ip, any destination ip, any service, any application, allow (defualt profile)

src-zone: Remote --> dst-zone: WiFi, any source ip, any destination ip, any service, any application, allow (defualt profile)

src-zone: WiFi --> dst-zone: External, any source ip, any destination ip, any service, any application, allow (defualt profile)

src-zone: any --> dst-zone: any, any source ip, any destination ip, any service, any application, deny (alert profile)

All traffic should be allowed to 3 zones. However, you will also need to allow the traffic for the IKE exchange between the Remote location's Peer IP address and your Peer IP address. I assume that these are both their external IP, and your external IP. For example:

We have an IPSEC tunnel between a server on our site and a server on a partner site. In order for the tunnel to be open, we have a rule that says:

src-zone: external --> dst-zone: external, src-IP: 156.x.x.x, dst-IP: 26.x.x.x, application <app list>, service application default, allow (default profile)

Even though our Server resides in the DMZ, the tunnel is built between the external interface on our PAN the external interface of the other site's ASA.

Anyway, you should be able to see what is being blocked in the traffic log with the block rule in place with an alert profile and the "log at session end" check box checked. This will allow you to see what the traffic flows are and what is being blocked, which will allow you define your rules to fit.

Re: What did I miss? Cross-zone/vpn traffic

Nathan.S — Tue, 16 Apr 2013 14:56:58 GMT

Here's the rules I have now, in this order:

src-zone: Wifi --> dst-zone: Remote, any source ip, any destination ip, any service, any application, allow (no profile)

src-zone: Remote --> dst-zone: Wifi, any source ip, any destination ip, any service, any application, allow (no profile)

src-zone: External --> dst-zone: External, <PAN public IP>, <Cisco Public IP>, any service, any application, allow (no profile)

src-zone: Wifi -> dst-zone: External, any source ip, any destination ip, any service, any application, allow (no profile)

src-zone: any -> dst-zone: any, any source ip, any destination ip, any service, any application, Deny (no profile)

From a client behind the Cisco, I can ping the PAN's inside IP (172.168.1.1) but not a client behind the PAN (172.168.1.10)

From a client behind the PAN, I can't ping the Cisco internal IP (10.5.1.1 which is pingable from internal) , or the client behind it (10.5.1.25)

tracert from behind the PAN goes outside (both by looking at IPs and from checking traffic log on the PAN)

traceroute from behind the Cisco never completes, but doesn't go outside.

Ping from behind the Cisco to the PAN logs as Remote-to-Wifi rule in the PAN.

Re: What did I miss? Cross-zone/vpn traffic

craymond — Tue, 16 Apr 2013 19:47:32 GMT

Under the system log, are you seeing the all of the vpn negotiations coming back as successful? Use the filter ( subtype eq vpn ) to see just the vpn traffic.

Also, in the traffic log are you seeing any denies? - use the filters ( app eq not-applicable ) or ( action eq deny ) to see the traffic that is being blocked.

Just in case you haven't reviewed yet - there is a document on IPSEC to ASA located here:

https://live.paloaltonetworks.com/docs/DOC-2579

Re: What did I miss? Cross-zone/vpn traffic

Nathan.S — Tue, 16 Apr 2013 21:53:00 GMT

All vpn attempts are successful.

Blocked traffic is one of two lines:

date/time, drop, from: External, to:External,source:<PAN public IP>, destination: <ISP DNS server>, to port: 53, application: not applicable, action: deny, rule: Deny-All, bytes: 84

date/time, drop, from: External, to:External,source:<PAN public IP>, destination: <Internal DNS server>, to port: 53, application: not applicable, action: deny, rule: Deny-All, bytes: 84

Traffic from Wifi, which should have gone across the VPN was being directed outside until I changed from the destination zone to the ip range of the far side of the VPN. Now it flows through the rules correctly, but still fails to ping.

Traffic from Remote across the vpn to client behind the PAN flows through the rules properly, but fails to ping either.

Thank you for the example/walk-through. I'd not found that one, only an older one dealing with PANos 3.x which was different enough to be not very helpful to this newb.

Accounting for minor differences in config, my ASA config matches that in the example.

Re: What did I miss? Cross-zone/vpn traffic

craymond — Thu, 18 Apr 2013 15:57:23 GMT

If you have your Proxy IDs set to /24s on PAN and /24s for ASA for protected traffic, and exempted NAT from the ASA side, I don't know why this is not functioning. Since the tunnel is up, and the traffic is supposed to be protected and allowed, I am at a loss as to why this isn't working for you. Hopefully someone else will chime in, or you might want to open an incident with TAC.

Re: What did I miss? Cross-zone/vpn traffic

Nathan.S — Thu, 18 Apr 2013 18:44:41 GMT

Set it /24 local and /16 remote. Reversed on the ASA.

I've got a ticket filed with support. Thanks for your help!

I'll post back up if I can find something specific to point to.

Re: What did I miss? Cross-zone/vpn traffic

Nathan.S — Fri, 19 Apr 2013 15:23:24 GMT

Worked with support for several hours yesterday.

Found that I'd left a PBF that was preventing local from following the rest of the rules I'd set in place. Doh!

Now I'm pinging from local to remote, but not from remote to local.

We're staring at config files on the ASA to see what we missed.

Re: What did I miss? Cross-zone/vpn traffic

craymond — Mon, 22 Apr 2013 18:19:06 GMT

You must have a little different routing setup then what I have dealt with. We did not need any PBF rules for our site to site VPN setup, and from all of the documentation that I have looked at, it never shows any PBF rules required to make it function. I did find that there were static routes needed on the ASA in the document that I linked to, but nothing more than that. Hopefully TAC will be able to discover what is missing. Sorry I could not help more!

Re: What did I miss? Cross-zone/vpn traffic

Nathan.S — Mon, 22 Apr 2013 18:24:23 GMT

PBF was part of my earlier attempts to get out-bound traffic working. Wasn't needed then, simply forgot to remove it.

Static routes are in place on the ASA. Matching the range behind the PAN.

At this point, I can see traffic in the log on the PAN as approved for passage, yet the ping does not complete.

I have my boss looking at the configs on the ASA to ensure they're properly configured.