SSH connection drops randomly

an.schall — Fri, 24 May 2019 14:26:08 GMT

An SSH connection to a particular server drops randomly (usually 20-60 seconds after login). Between the client and the server is a Palo Alto firewall with SSH decryption disabled.

What I tried so far

- regenerated ssh keys on the server
- added to server config:
ClientAliveInterval 30
ClientAliveCountMax 5
- added `ServerAliveInterval=10` to ssh command
- added `ServerKeepAlive=true` to ssh command
- tried various ssh clients

Nothing worked so far. Notice the debug3: send packet: type 80 and debug3: send packet: type 1 messages just at the moment before/ after the connection is dropped. The firewall logs the SSH session and the termination reason is "tcp-rst-from-client".

Here's a log of a representative SSH session, terminated ~15 seconds after login:

    user@nb ~/.ssh
    $ ssh -vvvvv -o ServerAliveInterval=10 root@10.1.1.2
    OpenSSH_7.9p1, OpenSSL 1.0.2r 26 Feb 2019
    debug2: resolve_canonicalize: hostname 10.1.1.2 is address
    debug2: ssh_connect_direct
    debug1: Connecting to 10.1.1.2 [10.1.1.2] port 22.
    debug1: Connection established.
    debug1: identity file /home/user/.ssh/id_rsa type -1
    debug1: identity file /home/user/.ssh/id_rsa-cert type -1
    debug1: identity file /home/user/.ssh/id_dsa type -1
    debug1: identity file /home/user/.ssh/id_dsa-cert type -1
    debug1: identity file /home/user/.ssh/id_ecdsa type -1
    debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
    debug1: identity file /home/user/.ssh/id_ed25519 type -1
    debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
    debug1: identity file /home/user/.ssh/id_xmss type -1
    debug1: identity file /home/user/.ssh/id_xmss-cert type -1
    debug1: Local version string SSH-2.0-OpenSSH_7.9
    debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u6
    debug1: match: OpenSSH_7.4p1 Debian-10+deb9u6 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
    debug2: fd 3 setting O_NONBLOCK
    debug1: Authenticating to 10.1.1.2:22 as 'root'
    debug3: hostkeys_foreach: reading file "/home/user/.ssh/known_hosts"
    debug3: record_hostkey: found key type ECDSA in file /home/user/.ssh/known_hosts:53
    debug3: load_hostkeys: loaded 1 keys from 10.1.1.2
    debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
    debug3: send packet: type 20
    debug1: SSH2_MSG_KEXINIT sent
    debug3: receive packet: type 20
    debug1: SSH2_MSG_KEXINIT received
    debug2: local client KEXINIT proposal
    debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
    debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
    debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
    debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
    debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
    debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
    debug2: compression ctos: none,zlib@openssh.com,zlib
    debug2: compression stoc: none,zlib@openssh.com,zlib
    debug2: languages ctos:
    debug2: languages stoc:
    debug2: first_kex_follows 0
    debug2: reserved 0
    debug2: peer server KEXINIT proposal
    debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
    debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
    debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
    debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
    debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
    debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
    debug2: compression ctos: none,zlib@openssh.com
    debug2: compression stoc: none,zlib@openssh.com
    debug2: languages ctos:
    debug2: languages stoc:
    debug2: first_kex_follows 0
    debug2: reserved 0
    debug1: kex: algorithm: curve25519-sha256
    debug1: kex: host key algorithm: ecdsa-sha2-nistp256
    debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
    debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
    debug3: send packet: type 30
    debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
    debug3: receive packet: type 31
    debug1: Server host key: ecdsa-sha2-nistp256 SHA256:xm9dqhSATd266dL4vmr1yMrdvJ7HFlMXx8tZSn4I7Pc
    debug3: hostkeys_foreach: reading file "/home/user/.ssh/known_hosts"
    debug3: record_hostkey: found key type ECDSA in file /home/user/.ssh/known_hosts:53
    debug3: load_hostkeys: loaded 1 keys from 10.1.1.2
    debug1: Host '10.1.1.2' is known and matches the ECDSA host key.
    debug1: Found key in /home/user/.ssh/known_hosts:53
    debug3: send packet: type 21
    debug2: set_newkeys: mode 1
    debug1: rekey after 134217728 blocks
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug3: receive packet: type 21
    debug1: SSH2_MSG_NEWKEYS received
    debug2: set_newkeys: mode 0
    debug1: rekey after 134217728 blocks
    debug1: Will attempt key: /home/user/.ssh/id_rsa
    debug1: Will attempt key: /home/user/.ssh/id_dsa
    debug1: Will attempt key: /home/user/.ssh/id_ecdsa
    debug1: Will attempt key: /home/user/.ssh/id_ed25519
    debug1: Will attempt key: /home/user/.ssh/id_xmss
    debug2: pubkey_prepare: done
    debug3: send packet: type 5
    debug3: receive packet: type 7
    debug1: SSH2_MSG_EXT_INFO received
    debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
    debug3: receive packet: type 6
    debug2: service_accept: ssh-userauth
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug3: send packet: type 50
    debug3: receive packet: type 51
    debug1: Authentications that can continue: publickey,password
    debug3: start over, passed a different list publickey,password
    debug3: preferred publickey,keyboard-interactive,password
    debug3: authmethod_lookup publickey
    debug3: remaining preferred: keyboard-interactive,password
    debug3: authmethod_is_enabled publickey
    debug1: Next authentication method: publickey
    debug1: Trying private key: /home/user/.ssh/id_rsa
    debug3: no such identity: /home/user/.ssh/id_rsa: No such file or directory
    debug1: Trying private key: /home/user/.ssh/id_dsa
    debug3: no such identity: /home/user/.ssh/id_dsa: No such file or directory
    debug1: Trying private key: /home/user/.ssh/id_ecdsa
    debug3: no such identity: /home/user/.ssh/id_ecdsa: No such file or directory
    debug1: Trying private key: /home/user/.ssh/id_ed25519
    debug3: no such identity: /home/user/.ssh/id_ed25519: No such file or directory
    debug1: Trying private key: /home/user/.ssh/id_xmss
    debug3: no such identity: /home/user/.ssh/id_xmss: No such file or directory
    debug2: we did not send a packet, disable method
    debug3: authmethod_lookup password
    debug3: remaining preferred: ,password
    debug3: authmethod_is_enabled password
    debug1: Next authentication method: password
    root@10.1.1.2's password:
    debug3: send packet: type 50
    debug2: we sent a password packet, wait for reply
    debug3: receive packet: type 52
    debug1: Authentication succeeded (password).
    Authenticated to 10.1.1.2 ([10.1.1.2]:22).
    debug1: channel 0: new [client-session]
    debug3: ssh_session2_open: channel_new: 0
    debug2: channel 0: send open
    debug3: send packet: type 90
    debug1: Requesting no-more-sessions@openssh.com
    debug3: send packet: type 80
    debug1: Entering interactive session.
    debug1: pledge: network
    debug3: receive packet: type 80
    debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
    debug3: receive packet: type 91
    debug2: channel_input_open_confirmation: channel 0: callback start
    debug2: fd 3 setting TCP_NODELAY
    debug3: ssh_packet_set_tos: set IP_TOS 0x48
    debug2: client_session2_setup: id 0
    debug2: channel 0: request pty-req confirm 1
    debug3: send packet: type 98
    debug2: channel 0: request shell confirm 1
    debug3: send packet: type 98
    debug2: channel_input_open_confirmation: channel 0: callback done
    debug2: channel 0: open confirm rwindow 0 rmax 32768
    debug3: receive packet: type 99
    debug2: channel_input_status_confirm: type 99 id 0
    debug2: PTY allocation request accepted on channel 0
    debug2: channel 0: rcvd adjust 2097152
    debug3: receive packet: type 99
    debug2: channel_input_status_confirm: type 99 id 0
    debug2: shell request accepted on channel 0
    Linux kali 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64

    The programs included with the Debian GNU/Linux system are free software;
    the exact distribution terms for each program are described in the
    individual files in /usr/share/doc/*/copyright.

    Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
    permitted by applicable law.
    Last login: Fri May 24 15:54:27 2019 from 10.128.108.39

[...random commands...]

debug3: send packet: type 80
    debug3: send packet: type 1
    Connection reset by 10.1.1.2 port 22

Re: SSH connection drops randomly

an.schall — Fri, 24 May 2019 15:09:08 GMT

One more information: I did a packet capture on the firewall to get more detailled information. I created all 4 packets capture stages (receive, transmit, drop and firewall). During an SSH session which gets disconnected as described above, the firewall also logs packets that are being dropped. Opening the pcap file that include the dropped packages, it becomes obvious that the capture includes only HSRPv2 type packets with Info: "Hello (state Active)" or "Hello (state Standby)". Could this be a layer 2 problem?

topic SSH connection drops randomly in General Topics

SSH connection drops randomly

Re: SSH connection drops randomly