Problem connecting SSH

[SOC_CSG]

I have a vpn configured (PA<->PA) to manage my FWs.

The problem is that when I open a ssh to the FW ip LAN (10.105.0.7), session ssh runs successfully and I can connect to the FW. But if I open ssh to the management ip 10.98.200.16 ssh remains frozen.

Looking at the log monitor, when i try the LAN ip i can see how the PA recognise the Application SSH, but trying with the management ip the FW is not recognising correctly and it shows INCOMPLETE.

Looking also at session browser when I run ssh session to the LAN ip, FW is not applying a Qos policy but is i try with management IP the FW is applying a Qos rule when it shouldnt do it.

Why this strange behavior between ssh in LAN ip and management IP.

I attached all the tests.

[HULK]

Hello Cos,

Incomplete means that either the three way TCP handshake did NOT complete or the three way TCP handshake did complete but there was no data after the handshake to identify the application. In other words, that traffic you are seeing is not really an application.

So to explain a little clearer, if a client sends a server a syn and the Palo Alto device creates a session for that syn, but the server never sends a SYN ACK in response back to the client, then that session would be seen as incomplete.

NOTE: From the above screenshot, it looks like, while you are trying to reach IP 10.98.200.16, only 78byte of data is being received at PAN FW.

Thanks

[HULK]

Hello Cos,

Could you please aslo check the real time session in the CLI by using 'show session all filter source IP_ADD_OF_THE_TESTING_PC(172.16.33.132)  destination IP_ADD_OF_THE_DESTINATION' (10.98.200.16).

>  If there is a session exist for the same traffic,  then please apply  CLI command PAN> show session id XYZ   >>>>>>>> to get detailed information about that session, i.e NAT rule, security rule, ingress/egress interface etc. I can see the session is not established here, that is why time out values shows only 5 seconds (pseudo session).

Thanks

[SOC_CSG]

Hi, this is the result. Any idea???

admin@GOLIAT1(active)> show session id 34480892

Session        34480892

        c2s flow:

                source:      172.16.33.132 [VPN]

                dst:         10.98.200.16

                proto:       6

                sport:       52478           dport:      22

                state:       INIT            type:       FLOW

                src user:    unknown

                dst user:    unknown

                qos node:    ethernet1/22.250, qos member N/A Qid 0

        s2c flow:

                source:      10.98.200.16 [TRANSITO]

                dst:         172.16.33.132

                proto:       6

                sport:       22              dport:      52478

                state:       INIT            type:       FLOW

                src user:    unknown

                dst user:    unknown

                qos node:    tunnel.1, qos member N/A Qid 0

        DP                            : 1

        index(local):                 : 182575

        start time                    : Tue Sep 16 18:05:01 2014

        timeout                       : 5 sec

        total byte count(c2s)         : 156

        total byte count(s2c)         : 0

        layer7 packet count(c2s)      : 2

        layer7 packet count(s2c)      : 0

        vsys                          : vsys1

        application                   : incomplete

        rule                          : Trafico VPN

        session to be logged at end   : True

        session in session ager       : False

        session synced from HA peer   : False

        layer7 processing             : enabled

        URL filtering enabled         : False

        session via syn-cookies       : False

        session terminated on host    : False

        session traverses tunnel      : True

        captive portal session        : False

        ingress interface             : tunnel.1

        egress interface              : ethernet1/22.250

        session QoS rule              : Capar-Jesus (class 4)

        tracker stage firewall        : Aged out

[HULK]

Hello COS,

Edited:

   total byte count(c2s)         : 156

        total byte count(s2c)         : 0 >>>>>>>>>>>>>>>>>>

        layer7 packet count(c2s)      : 2

        layer7 packet count(s2c)      : 0 >>>>>>>>>>>>>>>>>>>>>

You are trying to reach management IP of the PAN firewall...right..? Could you please share the service route configuration and how you are expecting this traffic to be routed. i can see traffic coming from Tunnel.1 interface and going out through ethernet1/22.250.

Thanks

[SOC_CSG]

I attached the service route config. I thought that to access by SSH you only needed to permit SSH in the Management interface.

[SOC_CSG]

Yes, its sending the trafiic to the interface1/22.250 and it should do it

admin@GOLIAT1(active)> test routing fib-lookup ip 10.98.200.16 virtual-router VR1 because the oowner of this ip is the PA. It shouldnt know the PA that it has this ip and not routing.

--------------------------------------------------------------------------------

runtime route lookup

--------------------------------------------------------------------------------

virtual-router:   VR1

destination:      10.98.200.16

result:           interface ethernet1/22.250

--------------------------------------------------------------------------------

admin@GOLIAT1(active)> show routing route destination 10.98.200.0/24 virtual-router VR1

flags: A:active, ?:loose, C:connect, H:host, S:static, ~:internal, R:rip, O:ospf, B:bgp,

       Oi:ospf intra-area, Oo:ospf inter-area, O1:ospf ext-type-1, O2:ospf ext-type-2

VIRTUAL ROUTER: VR1 (id 2)

  ==========

destination                                 nexthop                                 metric flags      age   interface          next-AS

10.98.200.0/24                              10.105.0.11                             20     A O2       1176268 ethernet1/22.250

total routes shown: 1

[HULK]

Yes, you are correct. But, how the PAN will forward this traffic to the management-interface, where no specific service (SSH) is is available on the service route . Could you please try to add a route there for destination IP 172.16.33.132 through ethernet1/22.250 ( service route).

Thanks

[SOC_CSG]

but the network 172.16.33.132 is reached through the tunnel. i think you mean destination 10.98.200.16 (management ip), but i think i will happen the same. Now its working like if i create the route that you want.

admin@GOLIAT1(active)> test routing fib-lookup ip 10.98.200.16 virtual-router VR1

--------------------------------------------------------------------------------

runtime route lookup

--------------------------------------------------------------------------------

virtual-router:   VR1

destination:      10.98.200.16

result:           interface ethernet1/22.250

[HULK]

Hello COS,

I can think about an alternative option, according to this situation( as a workaround).

If you have a layer-3 device which physically connected to the PAN FW's dataplane interface and an another connection to the management interface, we can send the SSH traffic coming through the VPN to router R1 and R1 will re-route the traffic to the management interface of the PAN. ( i have tested it in my PAN FW and it's working perfectly).

Hope this helps.

Thanks

[hshah]

Hi COS,

From thread I understood two things.

1. Packets are flowing through Data Plane.

2. SYN Is coming to firewall but, SYN/ACK is not being sent.

This conclude that, first packets are allowed and session is created. After that due to some reason packet is being dropped.

In this situation I would suggest you to do packet capture to check any drop packets.

Please refer following document for the same.

How to Run a Packet Capture

Once captures are configured execute following command.

show counter global filter packet-filter yes delta yes.

Again execute above command.

show counter global filter packet-filter yes delta yes.

And provide us output for second command and let us know if you see any drops.

Regards,

Hardik Shah

[SOC_CSG]

hi, i did several captures.

If i try to open ssh to the management IP, i only can see the SYN, not SYN/ACK is generated.

If i try to open ssh to the LAN IP, i oan see how SYN,SYN/ACK, and ACK are generated.

[hshah]

Hi COS,

Most likely its following bellow pattern.

1. SYN comes from Data Plane and hits Management Interface.

2. Now Management interface has different default gateway and its sending SYN/ACK out of Management Interface. And not on Data plane.

3. To verify same do packet capture on Management interface.

tcpdump filter "host 172.16.33.132"

4. Then execute following command to view capture.

view-pcap mgmt-pcap mgmt.pcap

5. If you see SYN/ACK in above output, then its confirm routing issue.

Let me know if this is helpful.

Regars,

Hardik Shah

[hshah]

Hi COS,

Long story short.

1. Login to CLI

2. put following command, its assumed source is 172.16.33.132.

tcpdump filter "host 172.16.33.132"

3. Execute Following command.

view-pcap mgmt-pcap mgmt.pcap

4. If you see SYN/ACK going out than its a assymetric routing issue.

Following document will help for packet capture.

https://live.paloaltonetworks.com/docs/DOC-4595

Regards,

Hardik Shah

[SOC_CSG]

If i use like filter 172.16.33.132 i cant see anything. I dont know what PA is doing with the traffic....

If, for example, i use like filter 10.98.200.16, i see several connections....