Forums

ssh and broken pipe

Hi All,

I am a paid user of PA. When I am trying to login over ssh I am getting a Broken Pipe error

Write failed: Broken pipe

I played around with the ServerAliveInterval timer but without success.

Any suggestions?

Thx

I just verified that ssh is working for me. I've never seen the error you speak of. Could you give more details as to when it is occurring? Like are you able to authenticate and get the prompt or is this when you attempt to connect...Perhaps you could provide your ssh event log to give an indication as to when in the process the trouble is happening.

Also, if you could tell us what technology you are using to connect. For instance I use Windows 7 x64 and PuTTY Development snapshot 2012-06-07:r9557

Working for me too!

Hi Roman,

the auth logs show your password being accepted... at what point do you see the Write failed message?

I've only seen this error when a server or client times out an idle connection, but that probably isn't the case if it happens straight away (and should only happen if the connection has been idle for too long).

To your knowledge, are you connecting via any sort of proxy server which could be closing your connection? For example, if you're connecting from a corporate server, some companies have extremely restrictive firewall configurations which can interfere with non-HTTP traffic. It would be surprising, since it appears that at least the authentication phase of SSH is completing, but it's worth a thought.

Also, there are a handful of ISPs which do some very dubious things with your network traffic. For example, Comcast once threatened to throttle or disconnect any encrypted traffic it didn't recognise (i.e. anything except HTTPS) in its overzealous war on P2P traffic. Haven't heard anything about that for awhile so hopefully they backed down.

Anyway, definitely looks like something killing the connection hard rather than being a graceful SSH error.

I'm certainly not going to defend Comcast, but for the record I use them for most of my traffic and have zero trouble with ssh/PA. However in case that does turn out to be the issue I will point out that my account with them is a BUSINESS account not a PERSONAL account, so if they did treat the traffic differently by account type it could still be a factor. I'm however of the opinion that they gave up the fight against P2P. They kind of didn't have a choice considering there are legitimate uses for the technology. But, that last statement is more of an opinion than fact, so perhaps I should have kept it to myself.

@a2j: Kept it to yourself? This is The Internet - I can't think a better place for statements that are more opinion than fact! (^_^) Still, it's interesting (and reassuring) to hear that Comcast are apparently not all bad.

Of course, like all good Internet Ranters I haven't had personal experience with them - I judge them based on the few people I know who use them and their public statements. I just dislike companies who give the impression that their customers are there to serve them, not the other way round.

The P2P issue was particularly absurd. I actually know the issue quite well because I worked on equipment for ISPs to reduce the costs of their customers' P2P traffic (without any negative impact on end users). I understand the pressure ISPs were/are under, and I can sympathise to an extent with ISPs who implement throttling (note I say sympathise not necessarily agree), but the ones who blocked P2P traffic entirely were bad enough. Extending this to throttling or blocking anything just because you can't identify the protocol involved is just... Hm, what's the antonym of "customer service"? Customer disservice?

Even without the complication that much P2P traffic infringes copyright law, there are still a lot of ISPs who talk about customers "abusing" their service because they have the audacity to use a high upload bandwidth. That's not abuse, that's just use. If they want to implement limits and throttling that's fine - provided they clearly advertise it before customers sign up. In the UK Virgin Media do it, and I'm one of their happy customers because I understood the limitations before I signed up, and because they at least offer me the ability to pay more to circumvent (most of) them.

@Cartroo: Remember I began with "I'm certainly not going to defend Comcast" and I hold to that statement.

Further I agree. I don't care in general how abusive companies are to their customers provided they are transparent about it. The marketplace will cleanse the garbage provided there is such transparency. I do make one aside however. In the case of monopolies/pseudo-monopolies then restriction is not only welcomed...it is necessary.

I'd argue that Comcast wouldn't exist if not for monopoly provisions. The problem is they gained such advantage from monopoly policy, that they are the best worst choice for certain products in my market.

Ok, guys, let's go back to my MAIN problem .... That I cannot ssh intp PA!

Here is a debug log of trying to login via ssh

====START=====

debug1: Authentications that can continue: publickey,password

debug1: Next authentication method: publickey

debug1: Offering RSA public key: /Users/roman/.ssh/id_rsa

debug1: Authentications that can continue: publickey,password

debug1: Offering DSA public key: /Users/roman/.ssh/id_dsa

debug1: Authentications that can continue: publickey,password

debug1: Next authentication method: password

romanbaumgaertner@ssh.pythonanywhere.com's password:

debug1: Authentication succeeded (password).

Authenticated to ssh.pythonanywhere.com ([23.21.200.247]:22).

debug1: channel 0: new [client-session]

debug1: Requesting no-more-sessions@openssh.com

debug1: Entering interactive session.

Write failed: Broken pipe

======END===========

By the way I tried to ssh from different systems (Mac, Linux, browser ssh client like FireSSH). In all the cases I am not able to get to the shell.

PS: and no Comcast talk .. PLEASE :-)

@romanbaumgaertner: Heh, sorry 'bout that. (^_^)

I found this and this whose symptoms appear similar. That said it sounds like they might only apply in chroot situations, but might be worth a glance in /var/log/auth.log on the server to see if the bad ownership or modes message is being produced.

Even if that isn't the issue (and there's a good chance it's a complete red herring), it sounds like this sort of failure might be caused when sshd decides it doesn't like something and closes the connection. Hopefully there might be some sort of log message which indicates why.

Both of those require the PA staff's help, however, so in the meantime if you're at home then you've more-or-less ruled out platform-specific issues by trying multiple systems, so perhaps it's related to the network you're on. Do you use a NAT home router by any chance? If so, perhaps try and reboot it, or look at the firewall rules to see if anything might be interfering?

If you can try from someone else's network connection (at work, maybe?) then that would be another interesting data point - if that suffers the same issue it's pretty good evidence it's server-side or something to do with your account.

Hi Cartroo,

Thanks for your response. The ssh does not work from home or from work. My colleague is able to ssh into PA on his Linux box. When I try use his box I am getting the favorite error "broken pipe".

My current guess is, that something is wrong with my account settings on the server side.

--Roman

OK, so maybe your home directory or ~/.ssh permissions got accidentally changed somehow. You haven't used any chmod or chown operations either in the shell or in Python have you?

Nope, nothing changed. I have no problems to ssh into my openshift accounts. It has only issues with PA form whatever system I am using.

Hmm. I suspect this is definitely something the PA staff would have to look into - sounds like something must have become messed up on your account. I'm sure that's what you thought to begin with, but it's useful to narrow down the source of the problem.

I think we've more-or-less ruled out anything your end or network-related. One minor point - when you and your friend were logging into PA from his Linux machine, were you both using his user account? If you were using your own account on the same machine then feasibly it could be due to differing SSH client configurations... Instinct tells me this is unlikely to be the case, however.

Just to clarify, on the PA side, have you ever been able to SSH in to your account? If it's been broken ever since your account was created then I wonder if there was a glitch in the account creation process. Also, can you use the web-based bash shell or is that broken too?

Good point Cartaroo - I believe you have to log out of the web interface, and then log back in to activate the ssh access. (That bit me.)

Just a quick note to say we're still investigating. We've ruled out a few things --

  • It's not that you need to log out and then back in again (that was a transitional thing when we added ssh access in the first place, and definitely doesn't affect romanbaumgaertner).
  • It's not the length of his username (a wild guess we thought we'd check out, but user we created user domanbaumgaertner and had no problems).
  • It's not a case thing, as his username is all lower case everywhere.

We'll keep investigating, I'll post back here if we have any more news.

Aha! Got it. It's a really weird bug in our ssh setup code. A specific file in romanbaumgaertner's sandbox was different to how it is in our test cases, and we don't handle it gracefully. I'll ping him with the details and post back here if it's not going to expose private information.

It shouldn't be too difficult to fix, anyway.

We've deployed a fix :-)

Hi Giles!

Great job! Sorry for causing the issue with the symlink :-( Now thinks are working fine and I can enjoy the ssh shell!

--Roman

No problem, I'm glad it's working now!

@romanbaumgaertner: I think you'll love it. I almost exclusively (when I'm not blogging here of course) use ssh for my PA stuff.

Oh, and one warning. I used to get D/C all the time, so I started using a short keep alive and now I can stay connected for days (if desired).