cancel
Showing results for 
Search instead for 
Did you mean: 

Who Me Too'd this topic

SSH connection drops randomly

L1 Bithead

An SSH connection to a particular server drops randomly (usually 20-60 seconds after login). Between the client and the server is a Palo Alto firewall with SSH decryption disabled.

What I tried so far

 - regenerated ssh keys on the server
 - added to server config:
 ClientAliveInterval 30
 ClientAliveCountMax 5
 - added `ServerAliveInterval=10` to ssh command
 - added `ServerKeepAlive=true` to ssh command
 - tried various ssh clients

Nothing worked so far. Notice the debug3: send packet: type 80 and debug3: send packet: type 1 messages just at the moment before/ after the connection is dropped. The firewall logs the SSH session and the termination reason is "tcp-rst-from-client".

Here's a log of a representative SSH session, terminated ~15 seconds after login:

    user@nb ~/.ssh
    $ ssh -vvvvv -o ServerAliveInterval=10 root@10.1.1.2
    OpenSSH_7.9p1, OpenSSL 1.0.2r  26 Feb 2019
    debug2: resolve_canonicalize: hostname 10.1.1.2 is address
    debug2: ssh_connect_direct
    debug1: Connecting to 10.1.1.2 [10.1.1.2] port 22.
    debug1: Connection established.
    debug1: identity file /home/user/.ssh/id_rsa type -1
    debug1: identity file /home/user/.ssh/id_rsa-cert type -1
    debug1: identity file /home/user/.ssh/id_dsa type -1
    debug1: identity file /home/user/.ssh/id_dsa-cert type -1
    debug1: identity file /home/user/.ssh/id_ecdsa type -1
    debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
    debug1: identity file /home/user/.ssh/id_ed25519 type -1
    debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
    debug1: identity file /home/user/.ssh/id_xmss type -1
    debug1: identity file /home/user/.ssh/id_xmss-cert type -1
    debug1: Local version string SSH-2.0-OpenSSH_7.9
    debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u6
    debug1: match: OpenSSH_7.4p1 Debian-10+deb9u6 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
    debug2: fd 3 setting O_NONBLOCK
    debug1: Authenticating to 10.1.1.2:22 as 'root'
    debug3: hostkeys_foreach: reading file "/home/user/.ssh/known_hosts"
    debug3: record_hostkey: found key type ECDSA in file /home/user/.ssh/known_hosts:53
    debug3: load_hostkeys: loaded 1 keys from 10.1.1.2
    debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
    debug3: send packet: type 20
    debug1: SSH2_MSG_KEXINIT sent
    debug3: receive packet: type 20
    debug1: SSH2_MSG_KEXINIT received
    debug2: local client KEXINIT proposal
    debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
    debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
    debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
    debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
    debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
    debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
    debug2: compression ctos: none,zlib@openssh.com,zlib
    debug2: compression stoc: none,zlib@openssh.com,zlib
    debug2: languages ctos:
    debug2: languages stoc:
    debug2: first_kex_follows 0
    debug2: reserved 0
    debug2: peer server KEXINIT proposal
    debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
    debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
    debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
    debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
    debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
    debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
    debug2: compression ctos: none,zlib@openssh.com
    debug2: compression stoc: none,zlib@openssh.com
    debug2: languages ctos:
    debug2: languages stoc:
    debug2: first_kex_follows 0
    debug2: reserved 0
    debug1: kex: algorithm: curve25519-sha256
    debug1: kex: host key algorithm: ecdsa-sha2-nistp256
    debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
    debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
    debug3: send packet: type 30
    debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
    debug3: receive packet: type 31
    debug1: Server host key: ecdsa-sha2-nistp256 SHA256:xm9dqhSATd266dL4vmr1yMrdvJ7HFlMXx8tZSn4I7Pc
    debug3: hostkeys_foreach: reading file "/home/user/.ssh/known_hosts"
    debug3: record_hostkey: found key type ECDSA in file /home/user/.ssh/known_hosts:53
    debug3: load_hostkeys: loaded 1 keys from 10.1.1.2
    debug1: Host '10.1.1.2' is known and matches the ECDSA host key.
    debug1: Found key in /home/user/.ssh/known_hosts:53
    debug3: send packet: type 21
    debug2: set_newkeys: mode 1
    debug1: rekey after 134217728 blocks
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug3: receive packet: type 21
    debug1: SSH2_MSG_NEWKEYS received
    debug2: set_newkeys: mode 0
    debug1: rekey after 134217728 blocks
    debug1: Will attempt key: /home/user/.ssh/id_rsa
    debug1: Will attempt key: /home/user/.ssh/id_dsa
    debug1: Will attempt key: /home/user/.ssh/id_ecdsa
    debug1: Will attempt key: /home/user/.ssh/id_ed25519
    debug1: Will attempt key: /home/user/.ssh/id_xmss
    debug2: pubkey_prepare: done
    debug3: send packet: type 5
    debug3: receive packet: type 7
    debug1: SSH2_MSG_EXT_INFO received
    debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
    debug3: receive packet: type 6
    debug2: service_accept: ssh-userauth
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug3: send packet: type 50
    debug3: receive packet: type 51
    debug1: Authentications that can continue: publickey,password
    debug3: start over, passed a different list publickey,password
    debug3: preferred publickey,keyboard-interactive,password
    debug3: authmethod_lookup publickey
    debug3: remaining preferred: keyboard-interactive,password
    debug3: authmethod_is_enabled publickey
    debug1: Next authentication method: publickey
    debug1: Trying private key: /home/user/.ssh/id_rsa
    debug3: no such identity: /home/user/.ssh/id_rsa: No such file or directory
    debug1: Trying private key: /home/user/.ssh/id_dsa
    debug3: no such identity: /home/user/.ssh/id_dsa: No such file or directory
    debug1: Trying private key: /home/user/.ssh/id_ecdsa
    debug3: no such identity: /home/user/.ssh/id_ecdsa: No such file or directory
    debug1: Trying private key: /home/user/.ssh/id_ed25519
    debug3: no such identity: /home/user/.ssh/id_ed25519: No such file or directory
    debug1: Trying private key: /home/user/.ssh/id_xmss
    debug3: no such identity: /home/user/.ssh/id_xmss: No such file or directory
    debug2: we did not send a packet, disable method
    debug3: authmethod_lookup password
    debug3: remaining preferred: ,password
    debug3: authmethod_is_enabled password
    debug1: Next authentication method: password
    root@10.1.1.2's password:
    debug3: send packet: type 50
    debug2: we sent a password packet, wait for reply
    debug3: receive packet: type 52
    debug1: Authentication succeeded (password).
    Authenticated to 10.1.1.2 ([10.1.1.2]:22).
    debug1: channel 0: new [client-session]
    debug3: ssh_session2_open: channel_new: 0
    debug2: channel 0: send open
    debug3: send packet: type 90
    debug1: Requesting no-more-sessions@openssh.com
    debug3: send packet: type 80
    debug1: Entering interactive session.
    debug1: pledge: network
    debug3: receive packet: type 80
    debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
    debug3: receive packet: type 91
    debug2: channel_input_open_confirmation: channel 0: callback start
    debug2: fd 3 setting TCP_NODELAY
    debug3: ssh_packet_set_tos: set IP_TOS 0x48
    debug2: client_session2_setup: id 0
    debug2: channel 0: request pty-req confirm 1
    debug3: send packet: type 98
    debug2: channel 0: request shell confirm 1
    debug3: send packet: type 98
    debug2: channel_input_open_confirmation: channel 0: callback done
    debug2: channel 0: open confirm rwindow 0 rmax 32768
    debug3: receive packet: type 99
    debug2: channel_input_status_confirm: type 99 id 0
    debug2: PTY allocation request accepted on channel 0
    debug2: channel 0: rcvd adjust 2097152
    debug3: receive packet: type 99
    debug2: channel_input_status_confirm: type 99 id 0
    debug2: shell request accepted on channel 0
    Linux kali 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64
   
    The programs included with the Debian GNU/Linux system are free software;
    the exact distribution terms for each program are described in the
    individual files in /usr/share/doc/*/copyright.
   
    Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
    permitted by applicable law.
    Last login: Fri May 24 15:54:27 2019 from 10.128.108.39
   
 [...random commands...]
 
 debug3: send packet: type 80
    debug3: send packet: type 1
    Connection reset by 10.1.1.2 port 22

Who Me Too'd this topic