topic Re: Problem connecting SSH in General Topics

Problem connecting SSH

SOC_CSG — Tue, 16 Sep 2014 14:24:39 GMT

I have a vpn configured (PA<->PA) to manage my FWs.

The problem is that when I open a ssh to the FW ip LAN (10.105.0.7), session ssh runs successfully and I can connect to the FW. But if I open ssh to the management ip 10.98.200.16 ssh remains frozen.

Looking at the log monitor, when i try the LAN ip i can see how the PA recognise the Application SSH, but trying with the management ip the FW is not recognising correctly and it shows INCOMPLETE.

Looking also at session browser when I run ssh session to the LAN ip, FW is not applying a Qos policy but is i try with management IP the FW is applying a Qos rule when it shouldnt do it.

Why this strange behavior between ssh in LAN ip and management IP.

I attached all the tests.

Re: Problem connecting SSH

HULK — Tue, 16 Sep 2014 15:47:34 GMT

Hello Cos,

Incomplete means that either the three way TCP handshake did NOT complete or the three way TCP handshake did complete but there was no data after the handshake to identify the application. In other words, that traffic you are seeing is not really an application.

So to explain a little clearer, if a client sends a server a syn and the Palo Alto device creates a session for that syn, but the server never sends a SYN ACK in response back to the client, then that session would be seen as incomplete.

NOTE: From the above screenshot, it looks like, while you are trying to reach IP 10.98.200.16, only 78byte of data is being received at PAN FW.

Thanks

Re: Problem connecting SSH

HULK — Tue, 16 Sep 2014 15:54:41 GMT

Hello Cos,

Could you please aslo check the real time session in the CLI by using 'show session all filter source IP_ADD_OF_THE_TESTING_PC(172.16.33.132) destination IP_ADD_OF_THE_DESTINATION' (10.98.200.16).

> If there is a session exist for the same traffic, then please apply CLI command PAN> show session id XYZ >>>>>>>> to get detailed information about that session, i.e NAT rule, security rule, ingress/egress interface etc. I can see the session is not established here, that is why time out values shows only 5 seconds (pseudo session).

Thanks

Re: Problem connecting SSH

SOC_CSG — Tue, 16 Sep 2014 16:07:53 GMT

Hi, this is the result. Any idea???

admin@GOLIAT1(active)> show session id 34480892

Session 34480892

c2s flow:

source: 172.16.33.132 [VPN]

dst: 10.98.200.16

proto: 6

sport: 52478 dport: 22

state: INIT type: FLOW

src user: unknown

dst user: unknown

qos node: ethernet1/22.250, qos member N/A Qid 0

s2c flow:

source: 10.98.200.16 [TRANSITO]

dst: 172.16.33.132

proto: 6

sport: 22 dport: 52478

state: INIT type: FLOW

src user: unknown

dst user: unknown

qos node: tunnel.1, qos member N/A Qid 0

DP : 1

index(local): : 182575

start time : Tue Sep 16 18:05:01 2014

timeout : 5 sec

total byte count(c2s) : 156

total byte count(s2c) : 0

layer7 packet count(c2s) : 2

layer7 packet count(s2c) : 0

vsys : vsys1

application : incomplete

rule : Trafico VPN

session to be logged at end : True

session in session ager : False

session synced from HA peer : False

layer7 processing : enabled

URL filtering enabled : False

session via syn-cookies : False

session terminated on host : False

session traverses tunnel : True

captive portal session : False

ingress interface : tunnel.1

egress interface : ethernet1/22.250

session QoS rule : Capar-Jesus (class 4)

tracker stage firewall : Aged out

Re: Problem connecting SSH

HULK — Tue, 16 Sep 2014 16:18:21 GMT

Hello COS,

Edited:

total byte count(c2s) : 156

total byte count(s2c) : 0 >>>>>>>>>>>>>>>>>>

layer7 packet count(c2s) : 2

layer7 packet count(s2c) : 0 >>>>>>>>>>>>>>>>>>>>>

You are trying to reach management IP of the PAN firewall...right..? Could you please share the service route configuration and how you are expecting this traffic to be routed. i can see traffic coming from Tunnel.1 interface and going out through ethernet1/22.250.

Thanks

Re: Problem connecting SSH

SOC_CSG — Tue, 16 Sep 2014 16:28:39 GMT

I attached the service route config. I thought that to access by SSH you only needed to permit SSH in the Management interface.

Re: Problem connecting SSH

SOC_CSG — Tue, 16 Sep 2014 16:40:34 GMT

Yes, its sending the trafiic to the interface1/22.250 and it should do it

admin@GOLIAT1(active)> test routing fib-lookup ip 10.98.200.16 virtual-router VR1 because the oowner of this ip is the PA. It shouldnt know the PA that it has this ip and not routing.

--------------------------------------------------------------------------------

runtime route lookup

--------------------------------------------------------------------------------

virtual-router: VR1

destination: 10.98.200.16

result: interface ethernet1/22.250

--------------------------------------------------------------------------------

admin@GOLIAT1(active)> show routing route destination 10.98.200.0/24 virtual-router VR1

flags: A:active, ?:loose, C:connect, H:host, S:static, ~:internal, R:rip, O:ospf, B:bgp,

Oi:ospf intra-area, Oo:ospf inter-area, O1:ospf ext-type-1, O2:ospf ext-type-2

VIRTUAL ROUTER: VR1 (id 2)

==========

destination nexthop metric flags age interface next-AS

10.98.200.0/24 10.105.0.11 20 A O2 1176268 ethernet1/22.250

total routes shown: 1

Re: Problem connecting SSH

HULK — Tue, 16 Sep 2014 16:54:23 GMT

Yes, you are correct. But, how the PAN will forward this traffic to the management-interface, where no specific service (SSH) is is available on the service route . Could you please try to add a route there for destination IP 172.16.33.132 through ethernet1/22.250 ( service route).

Thanks

Re: Problem connecting SSH

SOC_CSG — Tue, 16 Sep 2014 17:17:18 GMT

but the network 172.16.33.132 is reached through the tunnel. i think you mean destination 10.98.200.16 (management ip), but i think i will happen the same. Now its working like if i create the route that you want.

admin@GOLIAT1(active)> test routing fib-lookup ip 10.98.200.16 virtual-router VR1

--------------------------------------------------------------------------------

runtime route lookup

--------------------------------------------------------------------------------

virtual-router: VR1

destination: 10.98.200.16

result: interface ethernet1/22.250

Re: Problem connecting SSH

HULK — Tue, 16 Sep 2014 17:46:42 GMT

Hello COS,

I can think about an alternative option, according to this situation( as a workaround).

If you have a layer-3 device which physically connected to the PAN FW's dataplane interface and an another connection to the management interface, we can send the SSH traffic coming through the VPN to router R1 and R1 will re-route the traffic to the management interface of the PAN. ( i have tested it in my PAN FW and it's working perfectly).

Hope this helps.

Thanks

Re: Problem connecting SSH

hshah — Tue, 16 Sep 2014 17:59:27 GMT

Hi COS,

From thread I understood two things.

1. Packets are flowing through Data Plane.

2. SYN Is coming to firewall but, SYN/ACK is not being sent.

This conclude that, first packets are allowed and session is created. After that due to some reason packet is being dropped.

In this situation I would suggest you to do packet capture to check any drop packets.

Please refer following document for the same.

How to Run a Packet Capture

Once captures are configured execute following command.

show counter global filter packet-filter yes delta yes.

Again execute above command.

show counter global filter packet-filter yes delta yes.

And provide us output for second command and let us know if you see any drops.

Regards,

Hardik Shah

Re: Problem connecting SSH

SOC_CSG — Tue, 16 Sep 2014 18:43:04 GMT

hi, i did several captures.

If i try to open ssh to the management IP, i only can see the SYN, not SYN/ACK is generated.

If i try to open ssh to the LAN IP, i oan see how SYN,SYN/ACK, and ACK are generated.

Re: Problem connecting SSH

hshah — Tue, 16 Sep 2014 18:47:01 GMT

Hi COS,

Most likely its following bellow pattern.

1. SYN comes from Data Plane and hits Management Interface.

2. Now Management interface has different default gateway and its sending SYN/ACK out of Management Interface. And not on Data plane.

3. To verify same do packet capture on Management interface.

tcpdump filter "host 172.16.33.132"

4. Then execute following command to view capture.

view-pcap mgmt-pcap mgmt.pcap

5. If you see SYN/ACK in above output, then its confirm routing issue.

Let me know if this is helpful.

Regars,

Hardik Shah

Re: Problem connecting SSH

hshah — Tue, 16 Sep 2014 18:51:40 GMT

Hi COS,

Long story short.

1. Login to CLI

2. put following command, its assumed source is 172.16.33.132.

tcpdump filter "host 172.16.33.132"

3. Execute Following command.

view-pcap mgmt-pcap mgmt.pcap

4. If you see SYN/ACK going out than its a assymetric routing issue.

Following document will help for packet capture.

https://live.paloaltonetworks.com/docs/DOC-4595

Regards,

Hardik Shah

Re: Problem connecting SSH

SOC_CSG — Tue, 16 Sep 2014 19:05:00 GMT

If i use like filter 172.16.33.132 i cant see anything. I dont know what PA is doing with the traffic....

If, for example, i use like filter 10.98.200.16, i see several connections....

Re: Problem connecting SSH

hshah — Tue, 16 Sep 2014 19:09:45 GMT

HI COS,

It seems Management interface has not even seen SYN Packet. It proves SYN is dropped on Data plane.

Did you see any drop packet in capture, if NO than final thing is to do flow basics.

Follow bellow instructions for flow basics, if its not used with care than it can crash firewall. Let me know for any question.

Packet Capture, Debug Flow-basic and Counter Commands

Regards,

Hardik Shah

Re: Problem connecting SSH

bat — Tue, 16 Sep 2014 19:22:34 GMT

Hi COS

Could you please verify if there are any set permitted IP addresses defined under management interface settings?

And if the source IP from which you are doing SSH is in that permit IP address list ?

Hope it helps !

Thanks

Re: Problem connecting SSH

SOC_CSG — Tue, 16 Sep 2014 19:28:00 GMT

Yes, the ip 172.16.33.132 is allowed.

Ill do this debug tomorrow. Please confirm this config (filter), it could crash using this short filter????

debug dataplane packet-diag set filter on
debug dataplane packet-diag set filter match source 172.16.33.132 destination 10.98.200.16

debug dataplane packet-diag set log feature flow basic
debug dataplane packet-diag set log on

Re: Problem connecting SSH

hshah — Tue, 16 Sep 2014 19:36:12 GMT

Hi COS,

NOTE: I strongly encourage to contact TAC for flow basics, unless you are confortable.

Debug appears good to me. Make sure you disable logging once you try to access firewall. Use following command for it.

debug dataplane packet-diag set log off

Following command to view output of flow basics.

less dp0-log pan_packet_diag.log

Then following commands to disable everything

debug dataplane packet-diag set log off

debug dataplane packet-diag set filter off

debug dataplane packet-diag clear log log

Refer following guide for details.

Packet Based Troubleshooting - Configuring Packet Captures and Debug Logs

Regards,

Hardik Shah

Re: Problem connecting SSH

HULK — Tue, 16 Sep 2014 20:10:31 GMT

Hello COS,

Only for routing info, mgmt interface has the default gateway, but service route is for firewall services. For mgmt access, the mgmt interface will have to respond which will only happen if there is a default gateway. You have 2 options here.

For example: Option-1

If you want to keep Mgmt and physical interface in the same broadcast domain:

Mgmt interface IP-1.1.1.1/24

Data-plane Ethernet interface IP- 1.1.1.2/24

For example: Option-2

If you want to keep Mgmt and physical interface in 2 different broadcast domain:

Mgmt interface IP-2.2.2.1/24 Default gateway- R1(2.2.2.2)

Data-plane Ethernet interface IP- 1.1.1.2/24 Default gateway- R1 (1.1.1.2)

Thanks