Two firewalls, identical rules, different behaviour

[fjwcash]

As far as I've been able to determine, the configuration for the two firewalls (PA-500s) are identical (with different IPs/subnets obviously), but the way they handle blocked connections is very different.

On one firewall, a telnet to a blocked port just hangs until it eventually times out and the connection is dropped.

On the other firewall, a telnet to a blocked port connects and waits for input.  Once you type something and hit enter, the connection is immediately dropped.

For example, if you telnet to port 80, the first firewall gives the following:

$ telnet server.somedomain 80
Trying 1.2.3.4...
telnet: connect to address 1.2.3.4: Operation timed out
telnet: Unable to connect to remote host

While a telnet to port 80 on the other firewall gives the following:

$ telnet server.someotherdomain 80
Trying 1.2.3.5...
Connected to server.someotherdomain.
Escape character is '^]'.
GET /

<very long wait for it to timeout at this point>
Connection closed by foreign host.

The behaviour we want is the former, just hang the connection until it times out, or drop it right away.  The issue we're running into is that the provincial body that manages/monitors the Internet links for the schools is doing old-school port scanning of all our firewalls, and some of them are showing "successful connections" for random ports, and sending us security alerts.

My gut instinct is that this has to do with the way application filtering works, where the firewall accepts the first few packets but doesn't forward them through until it's identified the application, then decides whether to allow or block the connection.  Which is great, except why is the one firewall acting differently?  And how do we get them all to act the same way?

I'm in the process of comparing the rules between the firewalls, but so far, they are the same, in the same order (most were created from a template and are fairly generic) on each firewall.

[gwesson]

Try entering two newlines after the "GET /" to ensure your request is actually sent. Without that, it'll just hang waiting for the \r\n\r\n it expects to see.

I suspect that you're hitting different rules on the two firewalls. Try a security policy match test:

> test security-policy-match source 192.0.2.1 destination 1.2.3.4 protocol 6 from SRC-ZONE to DST-ZONE application web-browsing

Protocol 6 is TCP and web-browsing should match a generic GET request. 

Once you have that, if you can report back and include a screenshot of the rule it's matching or the text result from the CLI command above, it may be more clear as to what the problem is. Including your PAN-OS version and app version (show system info) can also help on this forum.

[fjwcash]

On firewall1, where it gives a timeout without ever connecting:

admin@PA500> test security-policy-match source <home-IP> destination <server-IP> protocol 6 from WAN to LAN application web-browsing destination-port 80

No matches.

On firewall2, where the telnet gives a connection prompt and let's me type "GET /" (with as many newlines as I want to type):

admin@PA500> test security-policy-match source <home-IP> destination <server-IP> protocol 6 from WAN to LAN application web-browsing destination-port 80

No matches.

Changing the destination zone to WAN also gives no matches on both firewalls.

Removing the source/destination zone and the application from the rule brings up our generic block-p2p rule on both firewalls (regardless of which port I use):

block-p2p-PAN {
        from any;
        source any;
        source-region none;
        to any;
        destination any;
        destination-region none;
        user any;
        category any;
        application/service [ kazaa/any/any/any gnutella/any/any/any soulseek/any/any/any direct-connect/any/any/any
 ares/any/any/any warez-p2p/any/any/any emule/any/any/any imesh/any/any/any bittorrent/any/any/any unknown-p2p/any/a
ny/any peerenabler/any/any/any 100bao/any/any/any fasttrack/any/any/any goboogy/any/any/any kugoo/any/any/any mute/a
ny/any/any openft/any/any/any poco/any/any/any soribada/any/any/any tesla/any/any/any thecircle/any/any/any xunlei/a
ny/any/any applejuice/any/any/any neonet/any/any/any winny/any/any/any azureus/any/any/any manolito/any/any/any pand
o/any/any/any allpeers/any/any/any gnunet/any/any/any ants-p2p/any/any/any fileswire/any/any/any generic-p2p/any/any
/any foxy/any/any/any winmx/any/any/any flashget/any/any/any fileguri/any/any/any totodisk/any/any/any perfect-dark/
any/any/any ];
        application/service(implicit) [ ssl/any/any/any web-browsing/any/any/any ];
        action deny;
        icmp-unreachable: no
        terminal no;
}

If I disable that rule, I get no matches for any combination of zones, no zones, no application, etc (not even the interzone/intrazone default rules).

All of the above was tested both without any current sessions active, and with an active telnet from <home-IP> to <server-IP> on port 80.  No difference in the results for either method.

[RobinClayton]

Have you checked the managemetn profile on the interfaces?

Rob

[fjwcash · Accepted solution]

Aha!

The interface protection profile is the same on both firewalls, pushed from Panorama.

BUT the Zone Protection Profile is different on the two.  On the one where the telnet connection just times out, the firewall is using the Zone Protection Profile pushed from Panorama where the Port Scan options are set to block-ip.

The other firewall, where the telnet would connect and you could type commands in before the connection would time out, had a local Zone Protection Profile with the Port Scan options set to alert.  Reverting the firewall to use the ZPP from Panorama made it act the same was as the other firewall.

🙂

Thanks for the nudge in the right direction.  Now we know where to look for differences on the other firewalls.  And now we get to go back through our tickets to figure out why we made that ZPP change on some firewalls.  😄