Forums

Hi hansel, MYSQL console error

ERROR 2003 (HY000): Can't connect to MySQL server on 'anywheredb.clt1czfvlzlc.us-east-1.rds.amazonaws.com' (111) Console closed.

Looks like the same problem as this post. Perhaps some routing or firewall issue within Amazon's network which is only affecting some regions?

This is very strange. I just started a MySQL console, so everything is presumably OK now. But as Cartroo says, someone else saw the same problem yesterday so it can't be a simple glitch. Looking at the AWS monitoring for that server, it's looks like it's been fine throughout.

The thing is, it can't be an inter-regional problem because if it was, lots of other things would have gone wrong. Same for a network glitch. I'll keep investigating, though. If anyone else sees the same problem, please leave a comment below.

I have the same problem with MySql console

Perhaps a rogue firewall rule which only affects the MySQL port somewhere, possibly only from certain subnets? (o_O)

I'm having the same problem as well.

Still have the same issues and been having these for a couple of days..

Thanks, Cartroo! -- that put me on the right track, and I think the problem is fixed now.

It looks like what happened was that the DB server concerned changed its IP address when Amazon bounced it for an upgrade. We have firewall rules in place to make sure that free users can access the database, and it looks like some of those didn't get updated with the new IP address. I created a new appropriately-configured free account, confirmed that I couldn't connect to the DB, then updated the rules, tried again, and confirmed that I could connect. So I think it should be fixed for everyone now.

If anyone's still seeing the problem, please do let me know!

You're welcome! It was just a stab in the dark, really. I always suspect firewall rules if issues selectively affect particular groups of users.

Mainly because if it's not a firewall rule somewhere, it often ends up being some really screwy routing issue involving things like BGP that I prefer to leave well alone, so I always hope it's the firewall issue... (^_^)

I'm having the same problem now: ERROR 2003 (HY000): Can't connect to MySQL server on 'anywheredb.#########.us-east-1.rds.amazonaws.com' (111) Console closed.

with the default database and with a new one.

I have the same problem... backend is down

I have the same problem with bash console, which opens in one line only. Pythonanywhere guys don't seem to manage all their issue since two days already :(

No probs here with MySQL consoles.

@compucen, looks like it's just you or you and a few others. I'm investigating.

@palagrassi, single-line consoles is usually because you have a zoom set on the page. Hit Ctrl-0 (that's control and the zero key) and reload the page. Let me know if that works.

Now it works thanks a lot :)

Mystery solved: Our MySQL server changed its IP address (it's a thing that AWS warned us could happen, but it's never happened before). This meant that free users whose network access is restricted were not configured to connect to the new IP address and so .

Same issue today :( ERROR 2003 (HY000): Can't connect to MySQL server on 'anywheredb.clt1czfvlzlc.us-east-1.rds.amazonaws.com' (111) is there something we can do when this happens (in case it's the IP address error reported above)?

I've had no MySQL server for hours now, and I'm not a free user.

Jim

This is really odd. Are you still seeing this problem? They're working fine for me from both free and paid accounts, on all console servers.

I've just signed up to test (free account) and am having this issue:

ERROR 2003 (HY000): Can't connect to MySQL server on 'anywheredb.clt1czfvlzlc.us-east-1.rds.amazonaws.com' (111)

Console closed.

I'm still getting this also. I've tried a new bash console.

Jim

OK, thanks for letting us know. Investigating.

Mine's started working again.

Jim

Excellent, thanks for confirming that!

So, it looks like what happened was:

• This morning at 6:15am, Amazon rebooted one of our database instances. We don't know why, and will put in a support call with them to find out why.
• The database outage disrupted operations on one of our file servers, which caused cascading outages throughout the system, taking down our site and a number of customers' sites.
• We were paged at 6:20am and logged in. The file server seemed to have recovered from the database outage, but the web and console servers connected to it were having NFS connection problems. We started a rolling reboot.
• By 6:45am everything appeared to be back to normal. Websites (including database-backed ones) were back online, as were MySQL consoles we started from the database tab for both free and paying accounts.

However, it looks like one of the effects of the DB outage was that it changed its IP address again. Most of our systems are now resilient to that. But running the mysql binary from a bash console (as opposed to using a MySQL console from the database tab) still goes though a system that needs restarting to pick up the IP address change. I've restarted that on all console servers, so everything should work now.

Things we need to do as a result of this:

• Find out why the DB instance was restarted, and if restarts like that are unavoidable for some Amazon maintenance purpose, to try to get them scheduled for times that aren't 6am on a Saturday morning so that we can handle them better. And ideally get advance warning so that we can pass that on to you, our customers.
• Find out why the IP address changed, and see if we can make the last few systems resilient to that.
• If we can't make them resilient, at least instrument them so that we get alerted when those systems are unresponsive so that we can fix them in a timely fashion.

Thanks to everyone for their patience with this problem.

Many thanks Giles.

I should just mention that I'm not running the mysql binary, rather connecting via Python 3.3 / SqlAlchemy / PyMySQL - not sure if that's relevant?

Jim

You're quite right, instead of

running the mysql binary from a bash console (as opposed to using a MySQL console from the database tab)

I should have said

accessing the database from a bash or Python console (as opposed to using a MySQL console from the database tab)

I still have the same issue whether I use the db tab or through a bash console.

I'm having this issue too

Same issue here.

@lacquerstash @guinunez @vivekannan

Have you got a 'Send feedback' link at the top? You could send a message to PA that way.

And are you all 'free' users, by the way?

Jim

We're investigating....

OK, looks like there may be a problem still affecting some free users. Sorry about that. Should have it fixed in 15 minutes.

OK, I think that's everything unblocked. Let me know if there are any more problems.

Working perfectly for me giles - thanks for keeping us informed!

thanks a lot, it is fixed now

Thanks for confirming! And sorry for the extended outage on this one.