sshd gives an error "Connection reset by peer" message

Issue

• sshd gives an error message into /var/log/messages after disconnecting a sshd-session properly.

sshd[14213]: Accepted password for npuser from ::ffff:192.168.153.69 port 3415 ssh2  
sshd(pam_unix)[14213]: session opened for user npuser by (uid=0)  
sshd(pam_unix)[14213]: session closed for user npuser  
sshd[14235]: fatal: Read from socket failed: Connection reset by peer

• As per above message, a sshd-session(PID:14213) seemd to be opened/closed properly, and this session is opened by running "scp" command to copy files to a remote server. The copy work is finished with no problem(all files are successfully copied), however, right after the sshd connection is disconnected, a fatal-error for another sshd session (PID:14235) appears. Although the scp command runs every two days, but the error message only happens sometimes.

• After capturing the tcpdump and analyzing it with tshark using the following command:

#tshark -r tcp.dump -R "tcp.flags.reset == 1" -z "proto,colinfo,ip.id,ip.id"

• We can see the following situation:

419918 32.811266 10.94.150.94 -> 10.94.150.12 TCP 36407 > ssh [RST, ACK] Seq=9804783 Ack=103454 Win=64088 Len=0 TSV=1972666552 TSER=27249498  ip.id == 0x3fc5  
443855 34.037567 10.94.150.94 -> 10.94.150.12 TCP 36406 > ssh [RST, ACK] Seq=15806719 Ack=163470 Win=64088 Len=0 TSV=1972667778 TSER=27250122  ip.id == 0xd534  
495587 36.268252 10.94.150.94 -> 10.94.150.12 TCP 36411 > ssh [RST, ACK] Seq=13504111 Ack=123566 Win=16022 Len=0 TSV=1972670009 TSER=27252994 ip.id == 0x75d1  
515993 36.957333 10.94.150.94 -> 10.94.150.12 TCP 36413 > ssh [RST, ACK] Seq=17896575 Ack=180222 Win=64088 Len=0 TSV=1972670698 TSER=27252266  ip.id == 0x10fe  
554154 42.691731 10.94.150.94 -> 10.94.150.12 TCP 36433 > ssh [RST] Seq=9080864 Win=0 Len=0  ip.id == 0x0000  
567515 43.281923 10.94.150.94 -> 10.94.150.12 TCP 36430 > ssh [RST, ACK] Seq=10148831 Ack=85214 Win=60016 Len=0 TSV=1972677024 TSER=27260061 ip.id == 0x0272

• Observe the "artificial" packet 554154:

554154 42.691731 10.94.150.94 -> 10.94.150.12 TCP 36433 > ssh [RST] Seq=9080864 Win=0 Len=0  ip.id == 0x0000

• When check the tcp.dump from the sender's part, we don't see this packet.

• It has no window, length or ip.id set. And it's not(!) sent by the sender, it's just coming out of nowhere (or, more precisely, from the network). The only fix till now is to repeatedly change the network hardware (switches etc.) until the bug is not reproducible.