SSH gets hung

Sometimes in one of our server the ssh gets hung for one particular id alone and if we remove out the /tmp/ssh- directory it is again working fine. Any possible reasons and solutions for this.