SCP transfer to one particular host is getting stalled

Latest response

Hi
We are trying to transfer an backup files to another host via scp and its getting stalled after some time. However from the same host when we transfer to other hosts there is no issues. There is no firewall in the path.

aj
Log in to join the conversation

Responses

james.radtke@siriusxm.com's picture Guru 6863 points
4 March 2015 3:57 PM

Hello Ajith,
Is the scp stalling at the same point every time? (specifically 2112) You may have an issue with your MTU and the switch hardware/config you are using.

Stack Overflow had a post on this topic
http://stackoverflow.com/questions/11985008/sending-a-large-file-with-scp-to-a-certain-server-stalls-at-exactly-2112-kb

aj Community Member 32 points
4 March 2015 7:19 PM

Hello James
No it is not stalling at 2112 kb. When we transfer batch files of say 30-50 gb, it gets stalled say 71%. We have this problem when we transfer to this one particular host.

Thanks

R. Hinton's picture Guru 6827 points
4 March 2015 6:50 PM

Ajith,

Just curious, is it only stalled, or does it just fail?

thanks

aj Community Member 32 points
4 March 2015 7:20 PM

Hello Hinton
Initially the speed of transfer decrases, like normally the transfer rate is 17 MB/s, but when the issue occurs it comes down to 9 Kb/s and it goes on for some time and then it finally stalls.

Thanks

os-unix's picture Active Contributor 146 points
4 March 2015 7:31 PM

Hi Ajith,

Hinton asked whether it stalls, but continues after sometime or it breaks after sometime.

Did you run top on the server that causes this stalling scp.

Did you open a Red Hat support case? Maybe support can find a configuration difference.

Kind regards,

Jan Gerrit Kootstra

aj Community Member 32 points
4 March 2015 7:53 PM

Dear Hinton
Once it stalls, it never continues. No we havent run top on the server. We will try that..

R. Hinton's picture Guru 6827 points
4 March 2015 8:59 PM

Hello Ajith, is there a chance the /etc/hosts.allow needs to be updated to allow that server or subnet for ssh/scp? just a thought...

R. Hinton's picture Guru 6827 points
4 March 2015 9:42 PM

Ajith, Please see the previous comments by myself and others...

Also, try running strace with the scp command such as: strace scp /path/to/file.xyz username@thatssystem:/path/to/target/location/ as well to see what that offers. Have a look at the logfiles as well on both servers at the very time you attempt the scp.

Are you using the typical Red Hat provided scp, and not a third party one such as Quest VAS?

Did scp ever work going to that system, or fail recently? Have you also attempted restarting the sshd service? Is the server joined to LDAP and unable to authenticate for proper users? If you are using ssh keys, is there any chance the permissions on the distant server for the .ssh and home directory for the user in question are fouled up? is the /etc/resolv.conf on the distant server set up properly to see dns and therefore the authentication server?

Is the time on the server your are doing an scp to proper? Is ntp running and or try an ntpdate -b [name-of-time-server]. Sometimes if a server is over five minutes off in time, then that can cause issues with authentication.

Please see the previous comments/questions from myself and others.

Thanks,

aj Community Member 32 points
7 March 2015 9:47 PM

Dear Hinton
Thanks for the inputs. We are using Openssh 4.3 that comes along with RHEL 5.8 Distribution. We restarted sshd on both nodes, Yes scp transfer was working perfectly before until we encountered this few weeks back. The Transfer starts without any problem , however after say 80% is transfered it stalls.

We did a strace on it and following is some of the sample outputs while stalling:-

rt_sigreturn(0x1) = -1 EINTR (Interrupted system call)
write(6, "o<\270\274fX\242N\217\361\222X/'\323\371\332\314\264=\351\0357\272\276\357\266\17\366\323_"..., 4096) = ? ERESTARTSYS (To be restarted)
--- SIGALRM (Alarm clock) @ 0 (0) ---
getpgrp() = 15822
ioctl(1, TIOCGPGRP, [15822]) = 0
PROVtracefile.070213.gz 25% 23MB 3.3KB/s - stalled -) = 168
rt_sigaction(SIGALRM, NULL, {0x2aba7afdbdb0, [], SA_RESTORER|SA_INTERRUPT, 0x2aba7ce1a2d0}, 8) = 0
alarm(1) = 0
rt_sigreturn(0x1) = -1 EINTR (Interrupted system call)
write(6, "o<\270\274fX\242N\217\361\222X/'\323\371\332\314\264=\351\0357\272\276\357\266\17\366\323_"..., 4096) = ? ERESTARTSYS (To be restarted)
--- SIGALRM (Alarm clock) @ 0 (0) ---
getpgrp() = 15822
ioctl(1, TIOCGPGRP, [15822]) = 0
PROVtracefile.070213.gz 25% 23MB 3.0KB/s - stalled -) = 168
rt_sigaction(SIGALRM, NULL, {0x2aba7afdbdb0, [], SA_RESTORER|SA_INTERRUPT, 0x2aba7ce1a2d0}, 8) = 0
alarm(1) = 0
rt_sigreturn(0x1) = -1 EINTR (Interrupted system call)
write(6, "o<\270\274f*X\242N\217\361\222X/'\323\371\332\314\264=\351\0357\272\276\357\266\17\366\323_"..., 4096) = ? ERESTARTSYS (To be restarted)
--- SIGALRM (Alarm clock) @ 0 (0) ---
getpgrp() = 15822
ioctl(1, TIOCGPGRP, [15822]) = 0
PROVtracefile.070213.gz

RL Community Member 56 points
8 March 2015 2:34 PM

Have you checked the obvious things such as having enough space/inodes on the destination server ?

aj Community Member 32 points
8 March 2015 9:54 PM

Yes Enough space is in the file system and inodes are also available. Most of the files in that file systems are big files consuming space, hence inodes are available.
Very Strange situation i have to say. Tried everything.

R. Hinton's picture Guru 6827 points
9 March 2015 1:59 PM

Ajith,

See this good tip from Tom Jones at this link https://access.redhat.com/discussions/1173853

  • Is there anything useful in the logs on the system you are going to via SSH?
  • Have a good look at /etc/nsswitch.conf on the target system, is it rational compared to other like systems?
  • Does the ip address on the system match dns?
    -- Are you using DHCP?
    -- Is the /etc/sysconfig/network file rational compared to other like systems?
  • Does the system reside in another network segment or DMZ where an external firewall may be a factor?
  • Also check this link at "ask.xmodulo.com" for some possibilities
  • Is the /etc/resolv.conf on the target system set to something proper (and pingable)
  • Are the default routes for the target system proper ( use netstat -rn and compare to another system)
  • Is the target system using LDAP, and if so, once on the system can you su - from one non-root user to another?
  • Are home drives being mounted from another location that is experiencing a delay?
  • Have you tried using ssh for a user whose account and home drive resides on the local system and not a remote file system?
  • Retry the ssh, but this time with the below (script -a will push all text on the screen to a text file such as below. replace "someuser" and "thatsystem" to what you use to ssh to that system)
script -a ~/ssh_verboseoutput.txt
ssh -vvv  someuser@thatsystem

At some point, type "ctrl x" to stop the output of the "script -a" above.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.