Have you made any firewall changes?

I would reboot you're Linode maybe the ssh service stopped for some reason.

Hello Kyre,

There have been no firewall changes. Moreover I have tried logging in from some other machine as well on a different network.

I have also tried rebooting the Linode, but it was of no good.

Lish - The Linode Shell

We've all done it at some point. Messed with your network settings and can't get into your Linode? No worries - Lish is your virtual console. Lish's primary function is to allow you access to your server's console, even if networking is disabled.

@ > Hello,

I contacted the linode support, who tried a few things and finally said, your linode must be misconfigured. They asked me to post here for answers.

Interesting

:!:

@kyrunner

Yes even I was little suprised by Linode support's answer.

And regarding the Lish shell. Yes it indeed works. I could use it instead of putty. I tried using it to troubleshoot the problem. However my major concern is, I cant login to using filezilla. I have constant code changes to be updated, which I am currently not able to do.

@ > @kyrunner

Yes even I was little suprised by Linode support's answer.

I have constant code changes to be updated, which I am currently not able to do.

I'm not understanding this.

I can use Linode's Lish instead of putty. So that isnt a problem.

However I cannot login using filezilla, to upload my web application files(jsps,css,images,js files) onto the server. I am using filezilla as the ftp, but its not let me login. I get a connection timeout issue.

@maxim@inovvorx.com:

I can use Linode's Lish instead of putty. So that isnt a problem.

However I cannot login using filezilla, to upload my web application files(jsps,css,images,js files) onto the server. I am using filezilla as the ftp, but its not let me login. I get a connection timeout issue.

What OS are you running? Centos,Ubuntu?

It is a Ubuntu 10.04 LTS

I meant my server OS is Ubuntu 10.04 LTS. I am using windows 7 with filezilla to try and connect.

@maxim@inovvorx.com:

It is a Ubuntu 10.04 LTS

What is the output of this command?

service vsftpd status

@Kyrunner: Thanks for the interest shown.

The output of the command mentioned is

vsftpd: unrecognized service

@maxim@inovvorx.com:

@Kyrunner: Thanks for the interest shown.

The output of the command mentioned is

vsftpd: unrecognized service

You are running this on your Linode server correct?

Don't run it on your windows machine run it on your Ubuntu server.

Yes Indeed. I used the Lish linode console to run this command. And this was the error reported.

@maxim@inovvorx.com:

Yes Indeed. I used the Lish linode console to run this command. And this was the error reported.

Well if you run this command (service vsftpd status), and its not recognized then that tells me ftp is not installed.

Yes ftp would not be installed on the server. However that should not stop filezilla on my windows 7 from connecting to the server machine isnt it?

@maxim@inovvorx.com:

Yes ftp would not be installed on the server. However that should not stop filezilla on my windows 7 from connecting to the server machine isnt it?

I'm confused how would you connect to a ftp server if the ftp server is not even installed? Filezilla is a client to connect to a ftp server (Filezilla windows 7) doesn't mean anything if the ftp server is not running.

You would use Filezilla or windows 7 to connect to a ftp server they are nothing more than a ftp client.

If he is using sftp, then ssh might be handling the server connection

My .02.

Edit: and indeed, that appears to be the case per the topic.

This is the help page of Linode http://library.linode.com/networking/fi … buntu-9.10">http://library.linode.com/networking/file-transfer/transfer-files-filezilla-ubuntu-9.10

It says nothing about installing a ftp server. Also when the linode was initially configured, no ftp server was explicitly installed. And also if ftp server was the problem, login via putty through SSH should have been possible isnt it?

Run this command

service ssh status

I get the output

ssh start/running, process 9577

@maxim@inovvorx.com:

I get the output

ssh start/running, process 9577

When you try to connect using putty you are putting in the wan ip of your Linode correct?

Yes… I have tried both ways… with the ip of the linode as well as the host name… putty waits for a while and gives a pop up Server unexpectedly closed the connection.

@maxim@inovvorx.com:

Yes… I have tried both ways… with the ip of the linode as well as the host name… putty waits for a while and gives a pop up Server unexpectedly closed the connection.

Run this command on your Linode and report back with its output.

grep -v "^#" /etc/ssh/sshd_config

Here is the output

IgnoreRhosts yes

RhostsRSAAuthentication no

HostbasedAuthentication no

PermitEmptyPasswords no

ChallengeResponseAuthentication no

X11Forwarding yes

X11DisplayOffset 10

PrintMotd no

PrintLastLog yes

TCPKeepAlive yes

AcceptEnv LANG LC_*

Subsystem sftp /usr/lib/openssh/sftp-server

UsePAM yes

==================================

Also the content of my sshd_Config file is as follows

Port 22

Use these options to restrict which interfaces/protocols sshd will bind to

ListenAddress ::

ListenAddress 0.0.0.0

Protocol 2

HostKeys for protocol version 2

HostKey /etc/ssh/sshhostrsa_key

HostKey /etc/ssh/sshhostdsa_key

Privilege Separation is turned on for security

UsePrivilegeSeparation yes

Lifetime and size of ephemeral version 1 server key

KeyRegenerationInterval 3600

ServerKeyBits 768

Logging

SyslogFacility AUTH

LogLevel INFO

I also posted to the support again and they said it would be better to reconfigure the SSH, by looking at at the ssh config file from a good system and making the necessary changes.

Please look for this line and tell me what is says

Change to no to disable tunnelled clear text passwords

PasswordAuthentication no

When you try to connect to your Linode using putty are you getting a error about public keys or no authentication method available.

No such error. When I reboot my linode I get the putty security warning about the rsa key.

The putty security alert as seen in the link below is all that I get, which I guess is normal.

http://library.linode.com/networking/using-putty

When I try via putty, all I can do is, enter the IP, following which it asks me for the login, which I mention as root, followed by the password prompt, which I enter. The cursor stops blinking there and after about 30 seconds I get the windows pop up alert saying "Server unexpectedly closed network connection"

@maxim@inovvorx.com:

No such error. When I reboot my linode I get the putty security warning about the rsa key.

The putty security alert as seen in the link below is all that I get, which I guess is normal.

http://library.linode.com/networking/using-putty

When I try via putty, all I can do is, enter the IP, following which it asks me for the login, which I mention as root, followed by the password prompt, which I enter. The cursor stops blinking there and after about 30 seconds I get the windows pop up alert saying "Server unexpectedly closed network connection"

Look for this line and report back.

Authentication:

LoginGraceTime 120

PermitRootLogin no

StrictModes yes

There isnt any such line which says disable rootlogin, however there is another which reads

PermitRootLogin yes

========================================

I am also pasting the entire sshd_config file below.

Package generated configuration file

See the sshd_config(5) manpage for details

What ports, IPs and protocols we listen for

Port 22

Use these options to restrict which interfaces/protocols sshd will bind to

ListenAddress ::

ListenAddress 0.0.0.0

Protocol 2

HostKeys for protocol version 2

HostKey /etc/ssh/sshhostrsa_key

HostKey /etc/ssh/sshhostdsa_key

Privilege Separation is turned on for security

UsePrivilegeSeparation yes

Lifetime and size of ephemeral version 1 server key

KeyRegenerationInterval 3600

ServerKeyBits 768

Logging

SyslogFacility AUTH

LogLevel INFO

Authentication:

LoginGraceTime 120

PermitRootLogin yes

StrictModes yes

RSAAuthentication yes

PubkeyAuthentication yes

AuthorizedKeysFile %h/.ssh/authorized_keys

Don't read the user's ~/.rhosts and ~/.shosts files

IgnoreRhosts yes

For this to work you will also need host keys in /etc/sshknownhosts

RhostsRSAAuthentication no

similar for protocol version 2

HostbasedAuthentication no

Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication

IgnoreUserKnownHosts yes

To enable empty passwords, change to yes (NOT RECOMMENDED)

PermitEmptyPasswords no

Change to yes to enable challenge-response passwords (beware issues with

some PAM modules and threads)

ChallengeResponseAuthentication no

Change to no to disable tunnelled clear text passwords

PasswordAuthentication yes

Kerberos options

KerberosAuthentication no

KerberosGetAFSToken no

KerberosOrLocalPasswd yes

KerberosTicketCleanup yes

GSSAPI options

GSSAPIAuthentication no

GSSAPICleanupCredentials yes

X11Forwarding yes

X11DisplayOffset 10

PrintMotd no

PrintLastLog yes

TCPKeepAlive yes

UseLogin no

MaxStartups 10:30:60

Banner /etc/issue.net

Allow client to pass locale environment variables

AcceptEnv LANG LC_*

Subsystem sftp /usr/lib/openssh/sftp-server

Set this to 'yes' to enable PAM authentication, account processing,

and session processing. If this is enabled, PAM authentication will

be allowed through the ChallengeResponseAuthentication and

PasswordAuthentication. Depending on your PAM configuration,

PAM authentication via ChallengeResponseAuthentication may bypass

the setting of "PermitRootLogin without-password".

If you just want the PAM account and session checks to run without

PAM authentication, then enable this but set PasswordAuthentication

and ChallengeResponseAuthentication to 'no'.

UsePAM yes

Hope this helps.

You haven't installed any blocking software like fail2ban have you?

It could be possible you have blocked your computer some how ether in the denied host file or through iptables or some kind of blocking software like fail2ban. try to connect using a different computer system.

Also run this and report back.

sudo grep -R "Received disconnect" log/

No installations. Also the denyhosts file is empty. And its just not my machine. I tried from a different network, and also from a different IP altogether. As a matter of fact I even got 2 people to try logging in from 2 different countries. And they reported the same error. Tried all ends and still clueless.

And regarding your command I get an error

grep: log/: No such file or directory

@maxim@inovvorx.com:

No installations. Also the denyhosts file is empty. And its just not my machine. I tried from a different network, and also from a different IP altogether. As a matter of fact I even got 2 people to try logging in from 2 different countries. And they reported the same error. Tried all ends and still clueless.

Looking at your sshd_config file everything looks good.

Any chance your not using the correct root password.

@maxim@inovvorx.com:

And regarding your command I get an error

grep: log/: No such file or directory

sudo grep -R "Received disconnect" /var/log/

sorry add /var/log

Firstly I am extremely grateful for your replies and patience. Thanks a lot.

Regarding the wrong password, I tried that as well. Thinking the password got reset for some reason, I tried reseting the password after powering off the linode, but no good. The same error persisted.

Regarding the Recieved disconnect here goes the out put

/var/log/auth.log:May 4 01:19:13 inovvorx sudo: root : TTY=hvc0 ; PWD=/root ; USER=root ; COMMA

ND=/bin/grep -R Recieved disconnect log/

/var/log/auth.log:May 4 01:19:22 inovvorx sudo: root : TTY=hvc0 ; PWD=/root ; USER=root ; COMMA

ND=/bin/grep -R Recieved disconnect log

/var/log/auth.log:May 4 01:20:07 inovvorx sudo: root : TTY=hvc0 ; PWD=/root ; USER=root ; COMMA

ND=/bin/grep -R Recieved disconnectlog/

/var/log/auth.log:May 4 01:23:05 inovvorx sudo: root : TTY=hvc0 ; PWD=/root ; USER=root ; COMMA

ND=/bin/grep -R Recieved disconnectlog/

/var/log/auth.log:May 4 01:24:34 inovvorx sudo: root : TTY=hvc0 ; PWD=/root ; USER=root ; COMMA

ND=/bin/grep -R Recieved disconnect /var/log/

@maxim@inovvorx.com:

Firstly I am extremely grateful for your replies and patience. Thanks a lot.

Regarding the wrong password, I tried that as well. Thinking the password got reset for some reason, I tried reseting the password after powering off the linode, but no good. The same error persisted.

Regarding the Recieved disconnect here goes the out put

/var/log/auth.log:May 4 01:19:13 inovvorx sudo: root : TTY=hvc0 ; PWD=/root ; USER=root ; COMMA

ND=/bin/grep -R Recieved disconnect log/

/var/log/auth.log:May 4 01:19:22 inovvorx sudo: root : TTY=hvc0 ; PWD=/root ; USER=root ; COMMA

ND=/bin/grep -R Recieved disconnect log

/var/log/auth.log:May 4 01:20:07 inovvorx sudo: root : TTY=hvc0 ; PWD=/root ; USER=root ; COMMA

ND=/bin/grep -R Recieved disconnectlog/

/var/log/auth.log:May 4 01:23:05 inovvorx sudo: root : TTY=hvc0 ; PWD=/root ; USER=root ; COMMA

ND=/bin/grep -R Recieved disconnectlog/

/var/log/auth.log:May 4 01:24:34 inovvorx sudo: root : TTY=hvc0 ; PWD=/root ; USER=root ; COMMA

ND=/bin/grep -R Recieved disconnect /var/log/

I'm running out of ideals

Is there anyway to clear the current SSH configuration and configure it all over again? I dont see any other option.

@maxim@inovvorx.com:

Is there anyway to clear the current SSH configuration and configure it all over again? I dont see any other option.

to be honest I don't think its a ssh issue… add a new user and try to log in

In your /etc/ssh/sshd_config, change

LogLevel INFO

to:

LogLevel Debug

then try and connect again. After it fails (or works!), look in /var/log/auth.log and copy and paste the relevant lines.

Don't forget to revert the LogLevel change after we're done.

Restart ssh after changing the loglevel using service ssh restart

The last few relevant lines of the auth.log file after setting it to debug is mentioned below.

May 4 10:09:37 inovvorx sshd[3781]: debug1: Bind to port 22 on ::.

May 4 10:09:37 inovvorx sshd[3781]: Server listening on :: port 22.

May 4 10:11:18 inovvorx sshd[3781]: debug1: Forked child 3785.

May 4 10:11:18 inovvorx sshd[3785]: Set /proc/self/oom_adj to -17

May 4 10:11:18 inovvorx sshd[3785]: debug1: rexec start in 5 out 5 newsock 5 pi

pe 7 sock 8

May 4 10:11:18 inovvorx sshd[3785]: debug1: inetd sockets after dupping: 3, 3

May 4 10:11:18 inovvorx sshd[3785]: debug1: Client protocol version 2.0; client

software version PuTTYRelease0.60

May 4 10:11:18 inovvorx sshd[3785]: debug1: no match: PuTTYRelease0.60

May 4 10:11:18 inovvorx sshd[3785]: debug1: Enabling compatibility mode for pro

tocol 2.0

May 4 10:11:18 inovvorx sshd[3785]: debug1: Local version string SSH-2.0-OpenSS

H_5.3p1 Debian-3ubuntu7

May 4 10:11:18 inovvorx sshd[3785]: WARNING: /etc/ssh/moduli does not exist, us

ing fixed modulus

May 4 10:11:28 inovvorx sshd[3785]: debug1: PAM: initializing for "root"

May 4 10:11:28 inovvorx sshd[3785]: debug1: PAM: setting PAM_RHOST to "117.192.

40.93"

May 4 10:11:28 inovvorx sshd[3785]: debug1: PAM: setting PAM_TTY to "ssh"

May 4 10:11:28 inovvorx sshd[3785]: Failed none for root from 117.192.40.93 por

t 11378 ssh2

I am still struggling on this one. If anyone can provide some insights, I would be really grateful.

This may not be of any use, but you can reset your sshd to the original configuration by removing it with apt-get purge openssh-server and re-installing with apt-get install openssh-server.

It sounds, however, like somehow your network connection is being cut off before you can transmit the password to the server. Something like fail2ban or denyhosts could do this, as mentioned earlier. You can see your current firewall rules with iptables -nvL.

Since you indicated you did not make any changes to the server configuration, you should also consider the possibility that it has been hacked.

@Vance, Thanks for you reply.

I purged the ssh config and tried re-installing it. However I got an error which read like this

/var/lib/dpkg/info/openssh-server.postinst: 459: cannot create /etc/ssh/sshd_config: Directory nonex

istent

dpkg: error processing openssh-server (–configure):

subprocess installed post-installation script returned error exit status 2

E: Sub-process /usr/bin/dpkg returned an error code (1)

And the problem still persists.

And the problem cant be due to any softwares installed, because the problem isnt for my machine/network alone. Everyone who tries to connect to the server from different locations report the same problem.

Also the output of the command iptables -nvL is as follows.

Chain INPUT (policy ACCEPT 4508K packets, 5346M bytes)

pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)

pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 3996K packets, 8098M bytes)

pkts bytes target prot opt in out source destination

OK, my statement about the network related to your Linode. If it is blocking hosts which attempt to connect via ssh, then it doesn't matter where the client is coming from - it will be blocked.

It appears that you have no firewall rules, so we can leave the network question aside for the moment.

It is very odd that attempting to re-install sshd did not work. I assume you were running these commands as root? To do so, you need to be logged in as root or put sudo in front of the command.

You can try to fix the problem (as root) with the commands mkdir -p /etc/ssh and apt-get –reinstall install openssh-server. This will create the missing directory that apt-get was complaining about and attempt to re-install sshd.

Hi all –

We are having a 100% identical issue as the original poster.

(1) We have a 512 box with the same Linode's deployment of Ubuntu 10.04 LTS.

(2) All the sudden, SSH started rejecting all login attempts.

(3) Lish does work.

Also, the last time we were able to login via SSH was exactly four days ago, just one day before before this thread was started. So it seems like the problem is more global as all symptoms are absolutely identical.

maxim, were you able to resolve the problem?

I also realized that we have a second LTS 10.04 Linode box – just checked it, and yes, it's having the same SSH problem, too!

So the issue is consistent across many (all?) Linodes with LTS 10.04

Can it be related to some kind of automatic update from Ubuntu?

I checked two 10.04 LTS Linodes, one an upgrade and the other a new install, and both seem to be working well. Have folks tried the -v option on the ssh client, to print out debugging info? In particular, the "remote software version" and "authentications that can continue" lines will be interesting.

Here's what I see, on the not-upgraded install,

rtucker@tremens:~$ ssh -v sapling.rocwiki.org
OpenSSH_5.9p1 Debian-5ubuntu1, OpenSSL 1.0.1 14 Mar 2012
debug1: Reading configuration data /home/rtucker/.ssh/config
debug1: /home/rtucker/.ssh/config line 10: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to sapling.rocwiki.org [2600:3c03::13:3b01] port 22.
debug1: Connection established.
debug1: identity file /home/rtucker/.ssh/id_rsa type -1
debug1: identity file /home/rtucker/.ssh/id_rsa-cert type -1
debug1: identity file /home/rtucker/.ssh/id_dsa type -1
debug1: identity file /home/rtucker/.ssh/id_dsa-cert type -1
debug1: identity file /home/rtucker/.ssh/id_ecdsa type -1
debug1: identity file /home/rtucker/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3p1 Debian-3ubuntu7
debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 50:fe:ae:71:fe:a4:2b:40:97:52:0e:dc:ef:e0:27:03
debug1: Host 'sapling.rocwiki.org' is known and matches the RSA host key.
debug1: Found key in /home/rtucker/.ssh/known_hosts:245
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/rtucker/.ssh/id_rsa
debug1: Trying private key: /home/rtucker/.ssh/id_dsa
debug1: Trying private key: /home/rtucker/.ssh/id_ecdsa
debug1: Next authentication method: password
rtucker@sapling.rocwiki.org's password: 
debug1: Authentication succeeded (password).
Authenticated to sapling.rocwiki.org ([2600:3c03::13:3b01]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LC_MESSAGES = en_US.UTF-8
debug1: Sending env LC_COLLATE = en_US.UTF-8
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LC_CTYPE = en_US.UTF-8
Linux sapling 3.0.4-linode38 #1 SMP Thu Sep 22 14:59:08 EDT 2011 i686 GNU/Linux
Ubuntu 10.04.4 LTS

And on the upgraded install,

rtucker@tremens:~$ ssh -v framboise
OpenSSH_5.9p1 Debian-5ubuntu1, OpenSSL 1.0.1 14 Mar 2012
debug1: Reading configuration data /home/rtucker/.ssh/config
debug1: /home/rtucker/.ssh/config line 10: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to framboise [2600:3c03::f03c:91ff:fe96:1dc9] port 22.
debug1: Connection established.
debug1: identity file /home/rtucker/.ssh/id_rsa type -1
debug1: identity file /home/rtucker/.ssh/id_rsa-cert type -1
debug1: identity file /home/rtucker/.ssh/id_dsa type -1
debug1: identity file /home/rtucker/.ssh/id_dsa-cert type -1
debug1: identity file /home/rtucker/.ssh/id_ecdsa type -1
debug1: identity file /home/rtucker/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3p1 Debian-3ubuntu7
debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA cd:91:1b:76:45:4c:90:c7:f7:c5:e3:0e:b0:33:a3:55
debug1: Host 'framboise' is known and matches the RSA host key.
debug1: Found key in /home/rtucker/.ssh/known_hosts:18
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Trying private key: /home/rtucker/.ssh/id_rsa
debug1: Trying private key: /home/rtucker/.ssh/id_dsa
debug1: Trying private key: /home/rtucker/.ssh/id_ecdsa
debug1: Next authentication method: keyboard-interactive
Password: 
Duo two-factor login for rtucker

<<redacted>>
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to framboise ([2600:3c03::f03c:91ff:fe96:1dc9]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LC_MESSAGES = en_US.UTF-8
debug1: Sending env LC_COLLATE = en_US.UTF-8
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LC_CTYPE = en_US.UTF-8
Success. Logging you in...
Linux framboise 3.0.4-linode38 #1 SMP Thu Sep 22 14:59:08 EDT 2011 i686 GNU/Linux
Ubuntu 10.04.4 LTS</redacted> 

No not yet… The issue still persists.. Thank God the problem is seen by other people as well. We were pretty stumped thinking what went wrong.

I'm looking for a post that I was reading awhile back that some kind of Malware was locking out ssh..I will report back when I find it.

Found the post.

http://www.linuxquestions.org/questions … ts-340366/">http://www.linuxquestions.org/questions/linux-security-4/failed-ssh-login-attempts-340366/

Edit: I'm not saying that this is the issue,but its worth looking into.

Ran tail /var/log/messages:

May 6 07:30:44 li374-19 kernel: VFS: Mounted root (ext3 filesystem) readonly on device 202:0.

May 6 07:30:44 li374-19 kernel: devtmpfs: mounted

May 6 07:30:44 li374-19 kernel: Freeing unused kernel memory: 388k freed

May 6 07:30:44 li374-19 kernel: Write protecting the kernel text: 5984k

May 6 07:30:44 li374-19 kernel: Write protecting the kernel read-only data: 1432k

May 6 07:30:44 li374-19 kernel: NX-protecting the kernel data: 3232k

May 6 07:30:44 li374-19 kernel: udev: starting version 151

May 6 07:30:44 li374-19 kernel: udevd (1036): /proc/1036/oom_adj is deprecated, please use /proc/10

36/oomscoreadj instead.

May 6 07:30:44 li374-19 kernel: Adding 262140k swap on /dev/xvdb. Priority:-1 extents:1 across:262

140k SS

May 6 07:30:44 li374-19 kernel: EXT3-fs (xvda): using internal journal

I don't know what it means exactly, but it looks like something happened to the kernel?

@cbrands

Not really sure what it is. However I havent seen anything of that sort on my linode logs.

@hoopycat, here is the debugging info i see with the -v.

[root@li40-97 ~]# ssh -vvvv root@19.x.x.x

OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: Applying options for *

debug2: ssh_connect: needpriv 0

debug1: Connecting to 19.x.x.x [19.x.x.x] port 22.

..and it times out

@cbrands:

debug1: Connecting to 19.126.125.19 [19.126.125.19] port 22.

..and it times out

That's a Ford IP address…. confused

@cbrands:

Ran tail /var/log/messages:

(…)

I don't know what it means exactly, but it looks like something happened to the kernel?

It looks like it rebooted at about 07:30. That would be "something" but probably not directly the cause of the problem. (Unless sshd isn't starting on boot…)

@sweh:

@cbrands:

debug1: Connecting to 19.126.125.19 [19.126.125.19] port 22.

..and it times out

That's a Ford IP address…. confused
Oh, you're trying to be cute and make people's life hard by obscuring details. See hoopycat's sig as to why this is a bad idea. Since you included a hostname in an earlier post, I'm gonna guess you really meant to use 96.126.125.19

You'll always get a timeout trying to use the Ford IP address 'cos it's not reachable from the internet.

@sweh:

You'll always get a timeout trying to use the Ford IP address 'cos it's not reachable from the internet.

I guess I accidentally tested the wrong IP. :shock: But here is what I see through: "cat /var/log/auth.log". It opens/closes all SSH connections:

May 6 14:09:01 li374-19 CRON[3101]: pam_unix(cron:session): session opened for user root by (uid=0)

May 6 14:09:01 li374-19 CRON[3101]: pam_unix(cron:session): session closed for user root

May 6 14:17:01 li374-19 CRON[3121]: pam_unix(cron:session): session opened for user root by (uid=0)

May 6 14:17:01 li374-19 CRON[3121]: pam_unix(cron:session): session closed for user root

May 6 14:39:01 li374-19 CRON[3146]: pam_unix(cron:session): session opened for user root by (uid=0)

May 6 14:39:01 li374-19 CRON[3146]: pam_unix(cron:session): session closed for user root

May 6 15:09:01 li374-19 CRON[3189]: pam_unix(cron:session): session opened for user root by (uid=0)

May 6 15:09:01 li374-19 CRON[3189]: pam_unix(cron:session): session closed for user root

@cbrands:

@sweh:

You'll always get a timeout trying to use the Ford IP address 'cos it's not reachable from the internet.

I guess I accidentally tested the wrong IP. :shock: But here is what I see through: "cat /var/log/auth.log". It opens/closes all SSH connections:

May 6 14:09:01 li374-19 CRON[3101]: pam_unix(cron:session): session opened for user root by (uid=0)

May 6 14:09:01 li374-19 CRON[3101]: pam_unix(cron:session): session closed for user root

May 6 14:17:01 li374-19 CRON[3121]: pam_unix(cron:session): session opened for user root by (uid=0)

May 6 14:17:01 li374-19 CRON[3121]: pam_unix(cron:session): session closed for user root

May 6 14:39:01 li374-19 CRON[3146]: pam_unix(cron:session): session opened for user root by (uid=0)

May 6 14:39:01 li374-19 CRON[3146]: pam_unix(cron:session): session closed for user root

May 6 15:09:01 li374-19 CRON[3189]: pam_unix(cron:session): session opened for user root by (uid=0)

May 6 15:09:01 li374-19 CRON[3189]: pam_unix(cron:session): session closed for user root

These are cron messages; nothing to do with ssh.

What happens when you 'ssh -v' with the correct IP address? Does it also fail when you try to log in as a non-root user?

@hoopycat:

What happens when you 'ssh -v' with the correct IP address? Does it also fail when you try to log in as a non-root user?

(1) Yes, it also fails (times out) when I try to log in as a non-root user.

(2) Below is the 'ssh -v' session with the correct IP address.

Hi all,

I do not use Linode but found this thread when searching exactly the same problem.

This is happened to me over the last few days on a number od servers, most of mine are ubuntu 10.04 and a couple of debian squeeze.

These servers have been fine and there is no auto update etc. I first started noticing a few days ago when one of my scheduled backups that uses ssh failed.

Its not just Linode :)

@DigitalDaz:

Its not just Linode :)

Daz please post here if you find a solution.

@cbrands:

@DigitalDaz:

Its not just Linode :)

Daz please post here if you find a solution.

I have a workaround using keys if you guys have access to some sort of console which I believe you do.

@Daz

Yes we do have access to the console. What is the workaround?

After 3 days, and no logical fix works, it's time to say your box is either fubar'd or compromised.

In either case, it's not time for a "workaround", it's time to cut your losses and spin up a fresh VPS.

No chance compromised, if it the same problem as mine.

I would have had to have about 10 different servers hacked, running a variety of different software and distros, and, some on different subnets. Probably more chance of me winning the lottery.

The workaround is simply to use puttygen to generate keys. Have a google around.

Then in your home directory if one does not already exist, create a directory .ssh

The key that goes on the server needs to be pasted into a file authorized_keys in the .ssh directory.

Then reconfigure putty or whatever you are using to use the keys to auth.

Mine goes straight in, no problems.

Its far more secure anyway, you can then go on to switch of password authentication later if you wish.

@Daz

Does it even solve the sftp issue as well?

I use Filezilla and Filezilla can use keys too.

Update:

We just hired a server admin to investigate this. He said that the PROBLEM is with the pam_unix module. The module is responsible for the password authentication, thus, the problem. It also explains why SSH keys still work.

He'll be fixing it for us tomorrow.

@Daz

We have a more irritating problem here :( The console that we use is browser specific and it does not allow Paste. Cant get the key pasted.

Try this then, this also worked for me for some strange reason.

At your console thing -

killall sshd

/usr/sbin/sshd

Then try ssh in, it took ages but worked, I don't think things are entirely borked here there is a timeout that can vbe altered because of that ^^^^ but I don't know which

@ Daz

How long did it take to login? I have increased the timeout to 10 mins… is that sufficient?

@Daz

Yes Indeed :-) finally the SSH login logged in successfully after quite sometime.

However the sftp doesnt seem to login after quite a long time. Is there anything different required to be done for the sftp.

@Daz

Omg… Finally ftp logged in too… after about 15 mins. You are the man!! I will owe you a great deal all my life. You were the medicine to my headache which was running all these days.

Also if possible can you point out what the possible reasons for the sudden change of behaviour could be? So that it helps the others facing this problem too.

Regards,

Maxim

Update: generated SSH keys and was able to login with the SSH keys!

@cbrands

How did u manage to paste the SSH keys using Lish? The paste doesnt work for me in any browsers: chrome,mozilla,ie,safari,opera.

Now your are ssh'd in paste the keys.

I too want to know what has caused this

@maxim@inovvorx.com:

@cbrands How did u manage to paste the SSH keys using Lish? The paste doesnt work for me in any browsers: chrome,mozilla,ie,safari,opera.

We didn't have to paste anything in Lish. The keys were generated on our Linode itself, and we copied them through Lish.

@Daz

The first time it did login. But it doesnt login anymore. It was late in the night yesterday when it worked the first time, and I thought I will do the key generation part today. However after a long wait, it doesnt seem to work now though. SSH and sftp doesnt login. Is it something like a one time login?

@Daz

I finally had no choice. I typed the public key manually. Beat that :) Took a solid 1 hour of patient effort for 2 people, one to read out and one typing, with a verification to cross check. Now it works like a charm. Thanks a lot.

Hello. I'm new to the forum, but found this post which applies to us. We have been using SFTP for about an year and no issues. All of a sudden we can't get in. Site is accessible via SSH, but SFTP is dead.

Any ideas? I went through some steps w/ linode support, but still no good. Please help! Thanks

@rsamee

Did u try this? This is what solved the problem for two of us facing this problem

The workaround is simply to use puttygen to generate keys. Have a google around.

Then in your home directory if one does not already exist, create a directory .ssh

The key that goes on the server needs to be pasted into a file authorized_keys in the .ssh directory.

Then reconfigure putty or whatever you are using to use the keys to auth.

Mine goes straight in, no problems.

Its far more secure anyway, you can then go on to switch of password authentication later if you wish.

Let me know if you need any help.

@maxim. Thank you for the response.

I haven't tried that yet. What's strange is:

1) worker in Pakistan = can no longer access

2) worker in Dominican Rep = can no longer access

3) worker in China = CAN access with no issue

I checked the firewall, nothing. No IPs blocked

How would I access my home directory now to paste those files, etc. Are you able to continue using SFTP via FileZilla? Thanks again!

@rsamee

We had the exact same problem. People from India, Pakistan, UAE couldnt access it. It was not an issue with the network from which you were trying to connect. As a matter of fact, we even had our linode migrated to another location on the same datacenter and that didnt solve the issue either. We are still not sure what caused the issue, but we do have a workaround for it. We use sftp too and it was blocked due to the problem.

And after you have done the steps mentioned above, you can use the key with any ftp such as filezilla or winscp to login.

And regarding your question how to access the console since you cant access ssh. I believe you are using linode. In the linode manager, you have a lish console under remote access section, which would work even if you are facing the issue.

If you need help add me on gmail chat at maxim@inovvorx.com and I will guide you over chat.

@rsamee

I didnt read that you had mentioned that you can access SSH (my access to SSH was also denied). If that is possible, you can create the file using commands over your console and paste the contents there.

Our server admin from Ukraine reinstalled ssh/pam. Everything works again. Victory!

Let me know if someone needs his services. He might be able to do it for like $25 per box.

Spot a problem with this one:

Someone joins a high value hosting company forum attracted to a topic about ssh issues and two days later suggests a sysadmin from Ukraine to fix the problem.

My apologies to said someone if he or she is legit, it's just that the above sentence, especially if read aloud, made me rofl.

@maxim

Strange thing is my developer in China can access it w/ no problems. I can access SSH also via putty, but filezilla and winscp won't connect. I was also reading online and someone said that filezilla servers are hosted on comcast, which could be causing the issue.. but I don't know about that.

@cbrands

I contacted my db guy already, but if you could get your server admin in touch w/ me that would be great! I can't afford to lose any more time as I have a live site deadline in less than 2 weeks! Please PM me his/her info.

Appreciate all of the help! Glad to know you can 'lean' on community members :)

@Azathoth

Well at least they FIXED the problem.

The rest of this thread is all about ignoring the actual problem and their exciting new workaround called "keys" (wow, who'd a thunk there was a better way of securing SSH). Nothing like putting a bandaid on your box and calling it done.

If someone manages to solve this can you please share it. I need the fix

@vonskippy:

@Azathoth

Well at least they FIXED the problem.

The rest of this thread is all about ignoring the actual problem and their exciting new workaround called "keys" (wow, who'd a thunk there was a better way of securing SSH). Nothing like putting a bandaid on your box and calling it done.

Who's putting bandaids on and calling it done then?

My server admin restarted SSH and that resolved the issue. Not sure why it needed to be restarted, but it worked for me!

I know others haven't had that luck, but all the best figuring this out if you don't have a resolution yet.

@Azathoth:

…My apologies to said someone if he or she is legit, it's just that the above sentence, especially if read aloud, made me rofl.

You're easily amused.

@DigitalDaz:

Who's putting bandaids on and calling it done then?

@Daz: his name is Stas and his email is gritsenko_stas at inbox.ru

He runs his own server/database support company. Mention ssh/pam in the subject line.

Tried restarting SSH and makes no difference now.

Found this post:

http://www.linuxquestions.org/questions … nt-369380/">http://www.linuxquestions.org/questions/linux-networking-3/ssh-login-works-but-sftp-doesnt-369380/

Not sure if you guys can make sense of that. It's over my head (n00b)

Thanks!

So, I found out why this happened to me and I managed to solve it.

I hope this will help out some of you out there.

1) You need to log on with the Lish console from the web interface. I tried to use Chrome for this and it gave me all sorts of problems

because for some reason I couldnt send commands like CTRL-C, Backspace or Escape or whatever .. a big problem if you mess up some commands,

or want to save a file you have edited. Which I needed to do.

2) Check the file /etc/ssh/sshd_config

Does it have the line "AllowUser [something]"?

Mine had. Mine had AllowUser gitosis.

3) Commented out that line

4) Restarted ssh with "service ssh restart".

edit: it also turned out my ISP had blocked all ssh connections out of the country. That certainly didn't help either. Had to get them to unblock the IP in addition to the solution mentioned above.

Worked.